01 July 2013

468. Kernel 3.10 on Debian

NOTE I: As of 3.10.2 the nvidia module will still not build.  I've also tried 3.10.5 and it also does not work.

NOTE II: I'm getting random slowdowns on my SL410 laptop with intel graphics. Not sure if it's the same issue as this: http://verahill.blogspot.com.au/2013/03/368-slow-mouse-and-keyboard-triggered.html
Once kworker shows up in top everything grinds to a slow crawl. I also notice that I never used 3.9 on that laptop, so the issue may be present there too.

There are several ways of building a kernel. The easiest (a purely subjective statement) is to use kernel-package i.e. make-kpkg. However, every now and again I see people writing that it's been deprecated.

Either way, start by doing
sudo apt-get install fakeroot build-essential ncurses-dev
mkdir ~/tmp
cd ~/tmp
wget https://www.kernel.org/pub/linux/kernel/v3.x/linux-3.10.tar.xz
tar xvf linux-3.10.tar.xz
cd linux-3.10/
cat /boot/config-`uname -r`>.config
make oldconfig
Timer tick handling 1. Periodic timer ticks (constant rate, no dynticks) (HZ_PERIODIC) (NEW) > 2. Idle dynticks system (tickless idle) (NO_HZ_IDLE) (NEW) 3. Full dynticks system (tickless) (NO_HZ_FULL) (NEW) choice[1-3]: Memory placement aware NUMA scheduler (NUMA_BALANCING) [N/y/?] (NEW) Simple CPU accounting cgroup subsystem (CGROUP_CPUACCT) [N/y/?] (NEW) Group CPU scheduler (CGROUP_SCHED) [N/y/?] (NEW) Automatic process group scheduling (SCHED_AUTOGROUP) [N/y/?] (NEW) Choose SLAB allocator 1. SLAB (SLAB) (NEW) > 2. SLUB (Unqueued Allocator) (SLUB) choice[1-2?]: Linux guest support (HYPERVISOR_GUEST) [N/y/?] (NEW) Timer frequency 1. 100 HZ (HZ_100) 2. 250 HZ (HZ_250) (NEW) 3. 300 HZ (HZ_300) > 4. 1000 HZ (HZ_1000) choice[1-4?]: Memory Hotplug (ACPI_HOTPLUG_MEMORY) [N/y/?] (NEW) AMD frequency sensitivity feedback powersave bias (X86_AMD_FREQ_SENSITIVITY) [N/m/?] (NEW) Kernel support for scripts starting with #! (BINFMT_SCRIPT) [Y/n/m/?] (NEW) InfiniBand media type support (TIPC_MEDIA_IB) [N/y/?] (NEW) Network Coding (BATMAN_ADV_NC) [N/y/?] (NEW) NETLINK: mmaped IO (NETLINK_MMAP) [N/y/?] (NEW) NETLINK: socket monitoring interface (NETLINK_DIAG) [N/m/y/?] (NEW) Dummy IRQ handler (DUMMY_IRQ) [N/m/y/?] (NEW) Generic on-chip SRAM driver (SRAM) [N/y/?] (NEW) ME Enabled Intel Chipsets (INTEL_MEI_ME) [N/m/y/?] (NEW) Block device as cache (BCACHE) [N/m/y/?] (NEW) Qualcomm Atheros AR816x/AR817x support (ALX) [N/m/y/?] (NEW) QLOGIC QLCNIC 83XX family SR-IOV Support (QLCNIC_SRIOV) [Y/n/?] (NEW) Realtek RTL8152 Based USB 2.0 Ethernet Adapters (USB_RTL8152) [N/m/?] (NEW) Atheros ath9k rate control (ATH9K_LEGACY_RATE_CONTROL) [N/y/?] (NEW) rt2800usb - Include support for rt55xx devices (EXPERIMENTAL) (RT2800USB_RT55XX) [N/y/?] (NEW) y Realtek RTL8188EE Wireless Network Adapter (RTL8188EE) [N/m/?] (NEW) IMS Passenger Control Unit driver (INPUT_IMS_PCU) [N/m/?] (NEW) Qualcomm Single-wire Serial Bus Interface (SSBI) (SSBI) [N/m/y/?] (NEW) Analog Devices ADT7310/ADT7320 (SENSORS_ADT7310) [N/m/y/?] (NEW) National Semiconductor LM95234 (SENSORS_LM95234) [N/m/?] (NEW) Nuvoton NCT6775F and compatibles (SENSORS_NCT6775) [N/m/y/?] (NEW) generic cpu cooling support (CPU_THERMAL) [N/y/?] (NEW) y ChromeOS Embedded Controller (MFD_CROS_EC) [N/m/y/?] (NEW) Silicon Laboratories 4761/64/68 AM/FM radio. (MFD_SI476X_CORE) [N/m/?] (NEW) System Controller Register R/W Based on Regmap (MFD_SYSCON) [N/y/?] (NEW) TI TPS65912 Power Management chip (MFD_TPS65912) [N/y/?] (NEW) Conexant cx25821 support (VIDEO_CX25821) [N/m/?] (NEW) Cypress firmware helper routines (CYPRESS_FIRMWARE) [N/m] (NEW) QXL virtual GPU (DRM_QXL) [N/m/?] (NEW) Apple infrared receiver (HID_APPLEIR) [N/m/?] (NEW) Enable USB persist by default (USB_DEFAULT_PERSIST) [Y/n/?] (NEW) USB-Wishbone adapter interface driver (USB_SERIAL_WISHBONE) [N/m/?] (NEW) USB Physical Layer drivers (USB_PHY) [N/y/?] (NEW) PXA 27x (USB_PXA27X) [N/m/?] (NEW) MARVELL PXA2128 USB 3.0 controller (USB_MV_U3D) [N/m/?] (NEW) LED Support for TI LP5562 LED driver chip (LEDS_LP5562) [N/m/?] (NEW) LED Camera Flash/Torch Trigger (LEDS_TRIGGER_CAMERA) [N/m/y/?] (NEW) y iSCSI Extentions for RDMA (iSER) target support (INFINIBAND_ISERT) [N/m/?] (NEW) Set system time from RTC on startup and resume (RTC_HCTOSYS) [Y/n/?] (NEW) Set the RTC time based on NTP synchronization (RTC_SYSTOHC) [Y/n/?] (NEW) RTC used to set the system time (RTC_HCTOSYS_DEVICE) [rtc0] (NEW) WIS GO7007 MPEG encoder support (VIDEO_GO7007) [N/m/?] (NEW) DesignWare USB2 DRD Core Support (USB_DWC2) [N/m/?] (NEW) pvpanic device support (PVPANIC) [N/m/y/?] (NEW) Reset Controller Support (RESET_CONTROLLER) [N/y/?] (NEW) XFS Verbose Warnings (XFS_WARN) [N/y/?] (NEW) Btrfs will run sanity tests upon loading (BTRFS_FS_RUN_SANITY_TESTS) [N/y/?] (NEW) Btrfs debugging support (BTRFS_DEBUG) [N/y/?] (NEW) EFI Variable filesystem (EFIVAR_FS) [N/m/y/?] (NEW) torture tests for RCU (RCU_TORTURE_TEST) [N/m/y/?] (NEW) Ring buffer startup self test (RING_BUFFER_STARTUP_TEST) [N/y/?] (NEW) Test functions located in the string_helpers module at runtime (TEST_STRING_HELPERS) [N/m/y] (NEW) CMAC support (CRYPTO_CMAC) [N/m/y/?] (NEW) SHA256 digest algorithm (SSSE3/AVX/AVX2) (CRYPTO_SHA256_SSSE3) [N/m/y/?] (NEW) y SHA512 digest algorithm (SSSE3/AVX/AVX2) (CRYPTO_SHA512_SSSE3) [N/m/y/?] (NEW) y Camellia cipher algorithm (x86_64/AES-NI/AVX2) (CRYPTO_CAMELLIA_AESNI_AVX2_X86_64) [N/m/y/?] (NEW) m Serpent cipher algorithm (x86_64/AVX2) (CRYPTO_SERPENT_AVX2_X86_64) [N/m/y/?] (NEW) m KVM legacy PCI device assignment support (KVM_DEVICE_ASSIGNMENT) [Y/n/?] (NEW) VHOST_SCSI TCM fabric driver (VHOST_SCSI) [N/m/?] (NEW)
make menuconfig

You can now enable any additional modules by navigating the menu structure. Note that most likely you don't have to enable anything in this step, but it can come in handy if there's a major transition (e.g. the way multimedia was handled changed between kernel 3.5 and 3.6) or if you want to enable a previously disabled option.

Then pick either method 1 or 2 below.

If you only want to compile modules that are currently in use (not a good idea if you want to use the same kernel on a range of computers, or have USB devices that aren't currently plugged in) you can do that by using make localmodconfig instead of make oldconfig. I wouldn't recommend it -- in most cases it won't make a faster kernel, and space and memory tends not to be much of in the way of issues these days.


Method 1. kernel-package
Below, change -j2 to -jX, where X is the number of cores in your CPU (not cores+1 or anything funny like that. See other posts on this blog for compilation performance tests)

sudo apt-get install kernel-package
make-kpkg clean
time fakeroot make-kpkg -j2 --initrd kernel_image kernel_headers
sudo dpkg -i ../linux-image-3.10.0_3.10.0-10.00.Custom_amd64.deb ../linux-headers-3.10.0_3.10.0-10.00.Custom_amd64.deb


Took 49 minutes on a 3-core AMD Athlon II, and used ca 7 Gb.

The files are shown below:
-rw-r--r--  1 me me 8.4M Jul  1 16:02 linux-headers-3.10.0_3.10.0-10.00.Custom_amd64.deb
-rw-r--r--  1 me me  32M Jul  1 16:00 linux-image-3.10.0_3.10.0-10.00.Custom_amd64.deb


Method 2. make deb-pkg

make clean
time make deb-pkg -j2
sudo dpkg -i ../linux-firmware-image_3.10.0-2_amd64.deb ../linux-headers-3.10.0_3.10.0-2_amd64.deb ../linux-libc-dev_3.10.0-2_amd64.deb ../linux-image-3.10.0_3.10.0-2_amd64.deb

It took ca 50 minutes, it used ca 7 Gb, and generated the following files:
-rw-r--r-- 1 me me 1.1M Jul  1 16:58 ../linux-firmware-image_3.10.0-2_amd64.deb
-rw-r--r-- 1 me me 9.7M Jul  1 16:58 ../linux-headers-3.10.0_3.10.0-2_amd64.deb
-rw-r--r-- 1 me me 458M Jul  1 17:13 ../linux-image-3.10.0_3.10.0-2_amd64.deb
-rw-r--r-- 1 me me 920K Jul  1 16:58 ../linux-libc-dev_3.10.0-2_amd64.deb

Note the size of the linux-image-3.10.0_3.10.0-2_amd64.deb package.


Difference:
There are a few differences. One method, the kpgk one, is supposedly 'deprecated', but it's been working fine in the past and will work fine in the future (at least for some time).

The make dev-pkg method also generates a much bigger image file -- by a factor of 15 or so.

Otherwise, the chief difference, from what I can see, is if you want to uninstall the kernel. Remove the libc-dev package requires you to downgrade the package using apt-get, by specifying a version e.g.

apt-cache policy linux-libc-dev
linux-libc-dev: Installed: 3.10.0-1 Candidate: 3.10.0-1 Version table: *** 3.10.0-1 0 100 /var/lib/dpkg/status 3.9.6-1~bpo70+1 0 100 http://ftp.iinet.net.au/debian/debian/ wheezy-backports/main amd64 Packages 100 http://ftp.debian.org/debian/ wheezy-backports/main amd64 Packages 3.2.46-1 0 500 http://ftp.iinet.net.au/debian/debian/ wheezy/main amd64 Packages 3.2.41-2+deb7u2 0 500 http://security.debian.org/ wheezy/updates/main amd64 Packages
sudo apt-get install linux-libc-dev=3.9.6-1~bpo70+1

I personally prefer the kernel-package approach.

467 wget and tor issue

Ever since I set up Tor on my debian workstation I've been having issues using wget:
wget https://www.kernel.org/pub/linux/kernel/v3.x/linux-3.10.tar.xz
--2013-07-01 11:32:35-- https://www.kernel.org/pub/linux/kernel/v3.x/linux-3.10.tar.xz Connecting to 127.0.0.1:9050... connected. Proxy tunneling failed: Tor is not an HTTP ProxyUnable to establish SSL connection.
For fun I also tried torify although I don't want to download the kernel via Tor:

torify wget https://www.kernel.org/pub/linux/kernel/v3.x/linux-3.10.tar.xz
--2013-07-01 11:36:20-- https://www.kernel.org/pub/linux/kernel/v3.x/linux-3.10.tar.xz Connecting to 127.0.0.1:9050... 11:36:20 libtorsocks(26463): connect: Connection is to a local address (127.0.0.1), may be a TCP DNS request to a local DNS server so have to reject to be safe. Please report a bug to http://code.google.com/p/torsocks/issues/entry if this is preventing a program from working properly with torsocks. failed: No such file or directory. Retrying.
Note that I DON'T want to use wget with Tor. I don't want to eat up bandwidth on the Tor network for stuff like this. When I use wget I want to use a direct connection.

I haven't configured /etc/wgetrc and so I was a bit surprised that this kept on happening.

The solution:
edit /etc/wgetrc and put
use_proxy=off

anywhere. And you're done.

28 June 2013

466. morph xyz -- python script to morph .xyz files

Rather naively I was hoping that by comparing two  molecule .xyz files and generating an average of them I would be able to conveniently generate a half-decent transition state guess.

Turns out that it's not quite as simple. However, I've written the software, so I might as well share it.

Note that it's written in python 2.7 (i.e. not python 3)

Run the script without arguments for help. General usage is
morphxyz -i 1.xyz 2.xyz -o morph.xyz


So here it is:

morphxyz:
#!/usr/bin/python

import sys

def getvars(arguments):
 switches={}

 version='0.1'
 
 try:
  if "-i" in arguments:
   switches['in_one']=arguments[arguments.index('-i')+1]
   switches['in_two']=arguments[arguments.index('-i')+2]
   print 'Input: %s and %s'% (switches['in_one'],switches['in_two'])
  else:
   arguments="--help";
 except:
  arguments="--help";
  
 try:
  if "-o" in arguments:
   switches['o']=arguments[arguments.index('-o')+1].lower()
   print 'Output: %s'% switches['o']
  else:
   arguments="--help";
 except:
  arguments="--help";

 try:
  if "-w" in arguments:
   switches['w']=float(arguments[arguments.index('-w')+1])
   print 'Weighting: %i'% switches['w']
  else:
   print 'Assuming no weighting'
   switches['w']=1.0;
 except:
  switches['w']=1.0;

 doexit=0
 try:
  if ("-h" in arguments) or ("--help" in arguments):
   print '\t\t bytes2words version %s' % version
   print '\t-i\t two xyz files to morph'
   print '\t-o\t output file'
   print '\t-w\t weight one structure vs the other (1=average; 0=start; 2=end)'
   print 'Exiting'
   doexit=1
 except:
  a=0 # do nothing
 if doexit==1:
  sys.exit(0)

 return switches

def getcmpds(switches):
 
 cmpds={}
 
 g=open(switches['in_one'],'r') 
 n=0
 xyz=[]
 atoms=[]
 
 for line in g:
  n+=1
  line=line.rstrip('\n')
  if n==1:
   cmpds['atoms_one']=int(line)
  elif n==2:
   cmpds['title_one']=line
  else:
   line=line.split(' ')
   line=filter(None,line)
   xyz+=[[float(line[1]),float(line[2]),float(line[3])]]
   atoms+=[line[0].capitalize()]
 cmpds['coords_one']=xyz
 cmpds['elements_one']=atoms
 
 g.close
 
 g=open(switches['in_two'],'r') 
 n=0
 xyz=[]
 atoms=[]
 
 for line in g:
  n+=1
  line=line.rstrip('\n')
  if n==1:
   cmpds['atoms_two']=int(line)
  elif n==2:
   cmpds['title_two']=line
  else:
   line=line.split(' ')
   line=filter(None,line)
   xyz+=[[float(line[1]),float(line[2]),float(line[3])]]
   atoms+=[line[0].capitalize()]
 cmpds['coords_two']=xyz
 cmpds['elements_two']=atoms
 g.close
 
 cmpds['w']=switches['w']
 
 return cmpds

def morph(cmpds):
 coords_one=cmpds['coords_one']
 coords_two=cmpds['coords_two']
 
 coords_morph=[]
 coords_diff=[]
 for n in range(0,cmpds['atoms_one']):
  morph_x=coords_one[n][0]+cmpds['w']*(coords_two[n][0]-coords_one[n][0])/2.0
  morph_y=coords_one[n][1]+cmpds['w']*(coords_two[n][1]-coords_one[n][1])/2.0
  morph_z=coords_one[n][2]+cmpds['w']*(coords_two[n][2]-coords_one[n][2])/2.0
  diff_x=coords_two[n][0]-coords_one[n][0]
  diff_y=coords_two[n][1]-coords_one[n][1]
  diff_z=coords_two[n][2]-coords_one[n][2]
  coords_morph+=[[morph_x,morph_y,morph_z]]
  coords_diff+=[[diff_x,diff_y,diff_z]]
 cmpds['coords_morph']=coords_morph
 cmpds['coords_diff']=coords_diff
 return cmpds

def genxyzstring(coords,element):
 x_str='%10.5f'% coords[0]
 y_str='%10.5f'% coords[1]
 z_str='%10.5f'% coords[2]
 
 xyz_string=element+(3-len(element))*' '+10*' '+\
 (8-len(x_str))*' '+x_str+10*' '+(8-len(y_str))*' '+y_str+10*' '+(8-len(z_str))*' '+z_str+'\n'
 
 return xyz_string

def writemorph(cmpds,outfile):
 g=open(outfile,'w') 
 h=open('diff.xyz','w')
 g.write(str(cmpds['atoms_one'])+'\n'+'\n')
 h.write(str(cmpds['atoms_one'])+'\n'+'\n')
 
 for n in range(0,cmpds['atoms_one']):
  coords=cmpds['coords_morph'][n]
  diffcoords=cmpds['coords_diff'][n]
  
  g.write(genxyzstring(coords, cmpds['elements_one'][n]))
  h.write(genxyzstring(diffcoords, cmpds['elements_one'][n]))
    
 g.close
 h.close
 return 0

if __name__=="__main__":
 arguments=sys.argv[1:len(sys.argv)]
 switches=getvars(arguments)
 cmpds=getcmpds(switches)
 
 if cmpds['atoms_one']!=cmpds['atoms_two']:
  print 'The number of atoms differ. Exiting'
  sys.exit(1)
 elif cmpds['elements_one']!=cmpds['elements_two']:
  print 'The types of atoms differ. Exiting'
  sys.exit(1)
  
 cmpds=morph(cmpds)
 success=writemorph(cmpds,switches['o'])
 if success==0:
  print 'Conversion seems successful'