Sunday 31 July 2011

Calibre, and Kindle, and books on the Calibre Webserver

Sad but true - I spent my friday night fiddling with PCs.

I've got a Kindle - an excellent reader - and a bunch of ebooks I've downloaded from Gutenberg. I already use Calibre to load the ebooks onto the kindle, but I liked the idea of accessing the library over my wifi network. Calibre has a built-in web server, on port 8080. (Also delete the calibre web username/password, and convert all your books to MOBI or LIT formats).

The Kindle browser is flawed though - it only connects over port 80 (HTTP) or 443 (S) :-(

I already had apache running on my main server (Giles) on port 80, so time for some tweaking:

I set up an alias for my server (in /etc/hosts/) :
127.0.0.1 localhost
127.0.1.1 giles

192.168.0.11 calibre.giles.local

# The following lines are desirable for IPv6 capable hosts
::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
192.168.0.11 is giles's static IP address. calibre.giles.local is my new virtual server name.

Added an apache conf file to use a virtual host (/etc/apache2/conf.d/calibre.conf):

ProxyPreserveHost On
ProxyPass / http://192.168.0.11:8080/
ProxyPassReverse / http://192.168.0.11:8080/
ServerName calibre.giles.local

This tells apache to route any traffic from calibre.giles.local to the calibre webserver (port 8080).

Now, using a browser on giles, I can see the calibre server by visiting http://calibre.giles.local

This works great, but only when I'm on giles - from any other machine, I would have to edit the 'hosts' file to add the DNS entry - and on the kindle that's not possible.

OK, more tweaking:

Installed dnsmasq (sudo apt-get install dnsmasq). This is a local DNS caching server - it caches the DNS records so it speeds up your browsing (fractionally) but more importantly, it also serves the local /etc/hosts file - so my new calibre.giles.local adress will be served to anyone using this DNS server.
Changed the /etc/resolv.conf:
nameserver 192.168.0.11
i.e. - get giles to use it's own dns server.

Changed /etc/dnsmasq.conf to add :
resolv-file=/etc/resolv2.conf
This avoids dnsmasq using itself to look up entries!

server=194.168.4.100
server=194.168.8.100
These are the two Virgin Media cable DNS servers (my ISP): I could also have used any public DNS servers.

Restarting dnsmasq (service dnsmasq restart) to pick up the changes, I can test this :
dig google.com 
gives me 9ms the first time, the 0ms the second - the cache is working.
dig calibre.giles.local 
shows 0ms - it is in the hosts file.

So, to make my entire network pick up the new DNS server: Login to my Apple Airport router, and go to IP (Internet) settings. Change the DNS list to have 192.168.0.11 first in the list.

All my other machines on the network use the airport router as the DHCP server and gateway/DNS proxy.

So:
Kindle connects to the wireless network - gets the DHCP address and a DNS server of 192.168.0.1 (the AirPort router).
Go to menu - experimental - web browser and type in "calibre.giles.local".
Kindle asks the airport router DNS for the address.
Airport router DNS asks Giles DNSmasq for the address. Found in local hosts file.

Kindle now displays the calibre web interface. You can search for books/authors, download and read any book (click on the MOBI file) :-)

Wednesday 20 July 2011

VNC problems in ubuntu?

I've been using VNC to remote access my two Ubuntu Natty Narwhal machines, and having terrible trouble - the VNC screen not updating, to controlling properly, etc.

It looks like the 3D and special effects play havoc with the VNC server - not displaying properly, etc.

Try using
gdmsetup
- the 'login settings' manager under the administration menu, or go to the top-right switch icon and and click System Settings, then 'Login Screen'.

Enter your admin password, then select 'Ubuntu Classic (No effects)' as the default session. Reboot or logout/login.

This changes ubuntu to use the standard gnome menubar and disables the 3d and animation effects.
I found that VNC worked much better with the classic menu and desktop effects off :-).

Monday 4 July 2011

Multiple webcams on ZoneMinder (Part 4)

Some success!

The USB 3.0 route turns out to be a slight misunderstanding. The key factors involved are :
1) Bandwidth required by each camera, and :
2) Number of USB 2.0 controllers.

It appears that USB 1.1, 2.0, and 3.0 subsystems are basically independent of each other.
A USB 2.0 motherboard has a usb 1.1 and a usb 2.0 controller. The *total* bandwidth for the usb2.0 controller is 480M, and any 2.0 devices share this bandwidth (whichever USB port or hub they are on). USB 1.1 devices similarly share a single 1.1 controller (often labelled a legacy controller).

The usb 3 hub I have seems to have a usb 3.0 hub + a usb 2.0 hub (+ probably a usb 1.1 hub).
Connecting two 2.0 devices (direct, of through the hub) shares the bandwidth of a single 2.0 controller on the USB 3.0 card.

Using the uvcvideo bandwidth quirks mode, I can connect two YUUV webcams to a single usb 2.0 controller - any more fail. This is true whether the port is a standard USB 2.0 hub in a 2.0 motherboard socket, or a USB 3.0 hub in a USB card.

To connect more cameras, add more 2.0 controllers. These can be in a USB 3.0 PCI-e card, or a plain USB 2.0 PCI card, or anything else. Apparently the USB 2.0 controllers included on the usb 3 chips are slightly better as they don't have any other demands on them and only have to deal with the usb 2 traffic.


NOTE : the PS3eye camera has a special 'bayer' mode that uses bandwidth more efficiently - if the driver can handle it. This allows windows users to plug in multiple PS3 eye cameras (see Code Laboratories or iPisoft) I haven't found a similar linux driver yet.

So far, I have been able to display 6 cameras simultaneously -
4 x Cheap USB 2.0 webcams - 640x480 at 25fps - YUYV, PAL
2 x PS3 eye cams - 640x480 at 12fps - YUYV, PAL - some breakup.

I have two cams on a USB 3.0 hub, to a USB 3.0 PCI-E card.
Two cams direct to motherboard controller ports.
Two ps3 eyes on USB 2.0 hub to a USB2.0 PCI controller card.

lsusb -t
davidr@hgwells:~$ lsusb -t
6-1:1.0: No such file or directory
/:  Bus 06.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 5000M
    |__ Port 3: Dev 3, If 0, Class=hub, Driver=hub/4p, 480M
        |__ Port 2: Dev 4, If 0, Class='bInterfaceClass 0x0e not yet handled', Driver=uvcvideo, 480M
        |__ Port 2: Dev 4, If 1, Class='bInterfaceClass 0x0e not yet handled', Driver=uvcvideo, 480M
        |__ Port 4: Dev 5, If 0, Class='bInterfaceClass 0x0e not yet handled', Driver=uvcvideo, 480M
        |__ Port 4: Dev 5, If 1, Class='bInterfaceClass 0x0e not yet handled', Driver=uvcvideo, 480M
/:  Bus 05.Port 1: Dev 1, Class=root_hub, Driver=ohci_hcd/2p, 12M
/:  Bus 04.Port 1: Dev 1, Class=root_hub, Driver=ohci_hcd/3p, 12M
/:  Bus 03.Port 1: Dev 1, Class=root_hub, Driver=ohci_hcd/8p, 12M
/:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=ehci_hcd/5p, 480M
    |__ Port 4: Dev 2, If 0, Class=vend., Driver=ov534, 480M
    |__ Port 4: Dev 2, If 1, Class=audio, Driver=snd-usb-audio, 480M
    |__ Port 4: Dev 2, If 2, Class=audio, Driver=snd-usb-audio, 480M
    |__ Port 5: Dev 3, If 0, Class=vend., Driver=ov534, 480M
    |__ Port 5: Dev 3, If 1, Class=audio, Driver=snd-usb-audio, 480M
    |__ Port 5: Dev 3, If 2, Class=audio, Driver=snd-usb-audio, 480M
/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=ehci_hcd/8p, 480M
    |__ Port 1: Dev 2, If 0, Class=hub, Driver=hub/4p, 480M
        |__ Port 1: Dev 3, If 0, Class='bInterfaceClass 0x0e not yet handled', Driver=uvcvideo, 480M
        |__ Port 1: Dev 3, If 1, Class='bInterfaceClass 0x0e not yet handled', Driver=uvcvideo, 480M
        |__ Port 2: Dev 4, If 0, Class='bInterfaceClass 0x0e not yet handled', Driver=uvcvideo, 480M
        |__ Port 2: Dev 4, If 1, Class='bInterfaceClass 0x0e not yet handled', Driver=uvcvideo, 480M
        |__ Port 4: Dev 5, If 0, Class=hub, Driver=hub/4p, 12M
            |__ Port 1: Dev 6, If 0, Class=HID, Driver=usbhid, 12M
            |__ Port 1: Dev 6, If 1, Class=HID, Driver=usbhid, 12M
            |__ Port 4: Dev 7, If 0, Class=hub, Driver=hub/4p, 12M
                |__ Port 2: Dev 8, If 0, Class=HID, Driver=usbhid, 12M
                |__ Port 2: Dev 8, If 1, Class=HID, Driver=usbhid, 12M
                |__ Port 3: Dev 9, If 0, Class=HID, Driver=wacom, 1.5M
davidr@hgwells:~$ 

Sunday 3 July 2011

Multiple webcams on ZoneMinder (part 3)

Alternative page for ubuntu AMD64 / zoneminder:

Webcam log:

Install zoneminder
davidr@hgwells:~$ sudo apt-get install zoneminder
Follow normal instructions for a zoneminder .deb installation:
sudo ln -s /etc/zm/apache.conf /etc/apache2/conf.d/zoneminder.conf
 sudo apache2ctl restart
 sudo chmod 4755 /usr/bin/zmfix
 zmfix -a
 sudo chown www-data.www-data /usr/share/zoneminder/temp
 sudo vi /etc/sysctl.conf 
Add the following lines at the end of the file:
kernel.shmall = 250000000
kernel.shmmax = 250000000
Pick up shared memory changes:
sudo sysctl -p
kernel.shmall = 250000000
kernel.shmmax = 250000000
I have 4 identical cameras plugged in. The order of video0-video3 may change if other cams, etc are added.
davidr@hgwells:~$ ls /dev/video*
/dev/video0  /dev/video1  /dev/video2  /dev/video3
Rather than use the video0 devices, there is a more fixed set of device links in /dev/v4l/by-id and /dev/v4l/by-path.
NOTE: there is a bug in zoneminder that the device name is limited. For some of my cameras, the by-id or by-path was truncated, and zoneminder couldn't find it.
davidr@hgwells:~$ ls /dev/video*
/dev/video0  /dev/video1  /dev/video2  /dev/video3
davidr@hgwells:~$ ls /dev/v4l/by-path/
pci-0000:03:00.0-usb-0:2.1:1.0-video-index0
pci-0000:03:00.0-usb-0:2.2:1.0-video-index0
pci-0000:03:00.0-usb-0:2.3:1.0-video-index0
pci-0000:03:00.0-usb-0:2.4:1.0-video-index0
So, to get around this problem, define my own paths:
davidr@hgwells:~$ sudo mkdir /cam
davidr@hgwells:~$ sudo chmod 777 /cam
davidr@hgwells:~$ cd /cam
davidr@hgwells:/cam$ ln -s /dev/v4l/by-path/pci-0000\:03\:00.0-usb-0\:2.1\:1.0-video-index0 c1
davidr@hgwells:/cam$ ln -s /dev/v4l/by-path/pci-0000\:03\:00.0-usb-0\:2.2\:1.0-video-index0 c2
davidr@hgwells:/cam$ ln -s /dev/v4l/by-path/pci-0000\:03\:00.0-usb-0\:2.3\:1.0-video-index0 c3
davidr@hgwells:/cam$ ln -s /dev/v4l/by-path/pci-0000\:03\:00.0-usb-0\:2.4\:1.0-video-index0 c4
davidr@hgwells:/cam$ ls
c1  c2  c3  c4
davidr@hgwells:/cam$ 
Use a terminal window with
tail -f /tmp/zmdc.log 
to check for any zoneminder problems.
Open firefox to http:\\hgwells.local\zm to get to the web interface.

Add a new monitor. On the source tab, set the source to:
device : /cam/c1
PAL
YUUV
width:320
height:240

I was able to get two cameras up so far: 3 and 4 fail with dmesg :
[ 159.696321] usb 3-2.4: Not enough bandwidth for altsetting 3
[ 172.716216] usb 3-2.3: Not enough bandwidth for new device state.

Good info on USB in linux

* claim for 5 PS3 eye cameras with one USB3.0 card *

Multiple webcams on ZoneMinder

Part 2:
Continuing the webcam saga (see part 1)

USB 1.1 and USB 2.0
I was incorrect in my earlier post. Modern motherboards usually have two usb controllers: a USB 2.0 controller and a 'legacy' USB 1.1 controller. Devices on each port are automatically switched to the right controller, so the problem of one usb 1 device slowing down the whole usb system does not occur.

I suspect something similar is implemented on most usb hubs - my USB 2.0 hub seems to cope fine with a mix of 1.1 and 2.0 devices.

Due to the wording of the USB 2.0 spec, some USB 1.1 webcams are labelled as USB 2.0 - despite the fact that they only connect at 12Mb (usb1.1). This is the case for two of my older webcams - a Targus and a Creative Live! cam both only connect at 1.1 speeds.

USB 3.0
Although Linux does have support for usb3 cards there is a bug in the current Ubuntu 'Natty Narwhal' kernel that means that USB 3.0 hubs may only be recognised as USB 2.0 hubs - you can see them using
lusub -t
connecting at 480M (usb 2.0).
This post suggested that a later version of the kernel would recognise the usb 3.0 hub properly.
I re-installed with the latest alpha version 11.10 (Oneiric Ocelot) which includes a later kernel. It now recognises the hub as 5000M.

Webcam USB Bandwidth
Some research on webcam USB bandwidth turned up a useful trick. Webcams request a whole load of bandwidth, usually more than they need.
Symptoms were fairly easy to spot: Opening one camera with
xawtv -v 1 -c /dev/video0
and at the same time opening /dev/video1 in another terminal led to
libv4l2: error turning on stream: No space left on device
VIDIOC_STREAMON - Unable to start capture: No space left on device
fps is set to 1/30
libv4l2: error turning on stream: Device or resource busy
VIDIOC_STREAMON - Unable to start capture: Device or resource busy

This post and this post gave me some useful pointers. The uvcvideo kernel module can be set to ignore the requested bandwidth, and to calculate the right bandwidth. Try:
sudo rmmod uvcvideo
sudo modprobe uvcvideo quirks=128
This will be reset every reboot. If this works, create the following file:
sudo vi /etc/modprobe.d/uvcvideo.conf 
containing the line:
options uvcvideo quirks=128

My Hardware setup
My new hardware - in case anyone wants to follow in my footsteps:
Transcend PCI Express Interface USB 3.0 Dual Expansion Card

Tsunami SSU34 100mm USB 3.0 4 Ports Hub

Webcam with Microphone for XP Vista PC Laptop MSN Skype

Output from lsusb :
davidr@hgwells:~$ lsusb -t
/:  Bus 04.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/2p, 5000M
    |__ Port 2: Dev 2, If 0, Class=hub, Driver=hub/4p, 5000M
/:  Bus 03.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/2p, 480M
    |__ Port 2: Dev 2, If 0, Class=hub, Driver=hub/4p, 480M
        |__ Port 1: Dev 3, If 0, Class='bInterfaceClass 0x0e not yet handled', Driver=uvcvideo, 480M
        |__ Port 1: Dev 3, If 1, Class='bInterfaceClass 0x0e not yet handled', Driver=uvcvideo, 480M
        |__ Port 2: Dev 4, If 0, Class='bInterfaceClass 0x0e not yet handled', Driver=uvcvideo, 480M
        |__ Port 2: Dev 4, If 1, Class='bInterfaceClass 0x0e not yet handled', Driver=uvcvideo, 480M
        |__ Port 3: Dev 5, If 0, Class='bInterfaceClass 0x0e not yet handled', Driver=uvcvideo, 480M
        |__ Port 3: Dev 5, If 1, Class='bInterfaceClass 0x0e not yet handled', Driver=uvcvideo, 480M
        |__ Port 4: Dev 6, If 0, Class='bInterfaceClass 0x0e not yet handled', Driver=uvcvideo, 480M
        |__ Port 4: Dev 6, If 1, Class='bInterfaceClass 0x0e not yet handled', Driver=uvcvideo, 480M
/:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=ohci_hcd/8p, 12M
/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=ehci_hcd/8p, 480M
    |__ Port 1: Dev 2, If 0, Class=hub, Driver=hub/4p, 480M
        |__ Port 4: Dev 3, If 0, Class=hub, Driver=hub/4p, 12M
            |__ Port 1: Dev 4, If 0, Class=HID, Driver=usbhid, 12M
            |__ Port 1: Dev 4, If 1, Class=HID, Driver=usbhid, 12M
            |__ Port 4: Dev 5, If 0, Class=hub, Driver=hub/4p, 12M
                |__ Port 2: Dev 6, If 0, Class=HID, Driver=usbhid, 12M
                |__ Port 2: Dev 6, If 1, Class=HID, Driver=usbhid, 12M
                |__ Port 3: Dev 7, If 0, Class=HID, Driver=wacom, 1.5M
davidr@hgwells:~$ lsusb
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 002 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 004 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 002: ID 05e3:0608 Genesys Logic, Inc. USB-2.0 4-Port HUB
Bus 003 Device 002: ID 2109:3431  
Bus 004 Device 002: ID 2109:0810  
Bus 001 Device 003: ID 05e3:0604 Genesys Logic, Inc. USB 1.1 Hub
Bus 003 Device 003: ID 090c:937b Feiya Technology Corp. Silicon Motion Camera
Bus 003 Device 004: ID 090c:937b Feiya Technology Corp. Silicon Motion Camera
Bus 003 Device 005: ID 090c:937b Feiya Technology Corp. Silicon Motion Camera
Bus 003 Device 006: ID 090c:937b Feiya Technology Corp. Silicon Motion Camera
Bus 001 Device 004: ID 046d:c049 Logitech, Inc. G5 Laser Mouse
Bus 001 Device 005: ID 05f3:0081 PI Engineering, Inc. Kinesis Integrated Hub
Bus 001 Device 006: ID 05f3:0007 PI Engineering, Inc. Kinesis Advantage PRO MPC/USB Keyboard
Bus 001 Device 007: ID 056a:0013 Wacom Co., Ltd Graphire 3 4x5