I'm wondering if there is any documentation for the serial protocol? I'm talking with someone about a potential used purchase as I've been considering adding support for the soundplane to my eurorack module... (the nw2s::b)
I've recently added support for a couple of USB devices including the monome and a gamepad. They went well, and now that I've got the USB host code basically worked out, I'd really like to include this guy for some easy computer-less soundplane-modular goodness.
I assume most folk work at the OSC or MIDI level, so that the only people who needed the raw USB protocol were the original programmers.
I'm also curious if there are any version differences between the different runs. The monome has some significant hardware capability changes from version to version. This seems a bit more consistent, but I thought I'd ask.
Thanks!
s
Looking at SoundplaneDriver.cpp in github, it looks like most everything is there.
Am I right in reading that basically all data is transferred as isochronous packets? This wil be my first isochronous device to build a driver for. Also looks like I'll also be unpacking the floats into 32 bit ints as best as I can to help the Due along a bit.
I'll let you know as soon as I have some more questions!
s
Okay, last one before bed. (Work tomorrow?!?!)
The usb driver unpacks raw FFT data and that is then translated by code running on the host that provides positioning data?
Can you point me to the class that is doing this base work?
thx.
s
Hi Scott,
It sounds like you are finding the right stuff. I took a look at SoundplaneDriver.h where the whole isoch buffer format is documented in the comments. This should be the main info you need. All the code that runs the Soundplane app is in the repo.
The low-level stuff in SoundplaneDriver gets the isoch data. SoundplaneModel::processCallback() is called on its own thread to get the latest frames.
My guess is that a Due will not have the CPU needed to run the algorithm I am currently using for touch detection. The emphasis was on speed and sensitivity, but CPU use is rather high. I am going to be working on the efficiency thing, so even if it doesn't work on your platform at first, I hope you push forward and see what happens, and we can work on this together over time.
The Soundplane's output protocol has not changed since the first Soundplanes were released.
Thanks for that overview. The callback is the glue I was missing.
The initial goal would just be to establish the communications link, receive the data, and unpack the FFT data without processing anything. Without any OS, this is usually most of the work. The soundplane is clearly an exception to that.
Looking through some of the processing code, I feel like initial goal would be to simply track pressure and position of a single point and I'm guessing I could take a few shortcuts doing that.
That will get me a lot more familiar with the data format so that I could move it to an STM32F4 which would triple the processing power, add floating point, and add a more robust instruction set - still not a macbook pro, but it would be dedicated hardware, so we'll see... step at a time.
Thanks,
Scott
I think you have exactly the right ideas in mind. Most of the difficulty in the touch tracking is doing multiple points. A single-point tracker could just look for the highest peak and would be very simple.
Hi Scott, sorry I'm a few weeks behind on noticing your questions.
You are correct: The Soundplane sends data on Isochronous endpoints, one for each half of the touch surface, with a sequence number to allow pairing of matching packets. Note that the USB Specification forbids a Device from sending Isochronous data until after the Host has selected an Alternate Interface. So, your code will first have to navigate the Descriptors and send the appropriate commands to select Alternate Interface #1. Alternate Interface #0 has no endpoints, while #1 has two.
Upon successful selection of Alternate Interface #1, the Soundplane powers up its analog circuits and the "power" LED lights up. You'll know that your commands are working if you see the LED lit.
At this point, you'll have 12-bit data coming in as highly-packed transfers. The source code that you already have shows how to decode this data. It's fixed-point in the USB packets, so you won't need floating point unless that's how you want to process the touches. A good test is to display or otherwise visualize the 2D grid of pressures as if it were a video frame, but perhaps that's not easily done with your platform.
There are more details, but hopefully the source code is precise enough since it is what everyone is using.
Free free to ask more questions, and I'll try to follow this thread.
Brian
Hi Scott, can we get an update on your progress? I'm probably not the only one who is interested to read about what you were able to get working, and whether you've upgraded to the STM32F4 and floating point.
Hi @rsdio and @scottwilson
Id also be interested if any progress has been made on this...
Im interested in getting the SP to directly connect to Axoloti, which is based on STMF4 using Chibios + STM host lib (etc), I helped get the USB host midi working on Axoloti, so have some experience on this.
Ive now got a full debugging environment for axoloti, and I also have an F7 discovery board here,
so im setup to go... the only thing i don't have (as outrageously expensive) is a USB analyser
(Im also pretty familiar with the soundplane software)
I see there are two tasks:
a) writing USB level
I've look at the SP driver, doesn't look particular complex, main new thing for me is getting iso working on STM. (Ive done using libusb before, but not the STM lib)
b) touch tracker
Whilst the SP software may be portable, having done a bit of work on the STM32F4 now, Id be concerned that its going to be a bit heavy.
and of course there is potential to use the cortex instructions to significantly increase performance.
(this is no criticism to the SP code, code for a modern computer makes different compromises for when we come to footprint, efficiency, flexibility, readability ... as it has much more leeway)
as Ive already said to Randy, I'm happy doing (a) but to do a reasonable job of (b) is difficult, I could probably getting it tracking single touches ok, but thats useless (to me) , and going beyond that gets complex very quickly.
so I think (b) is where ML could really help out... as I assume you will need this for your eurorack module.. and the STM32F4 (or F7, which is compatible) is the most obvious choice.
perhaps we could collaborate?
I plan to keep the touch tracker code open source. It should be easy to port. I'm certainly up for helping, as much as my time permits. There's also a Linux port in the works using libusb. I have no idea if an STM32F4 will be capable of running the touch detector, but let me finish it on Intel and then I can help try to make that happen.
@thetechnobear
Of the many things you have to tackle, you could get a start by looking for USB Host isochronous examples, such as would allow you to connect a USB Audio interface and stream audio. If you can get that going on your STM32F4, then you could probably easily host the Soundplane with a few modifications. Actually, the Soundplane requires much less complex support than an audio interface, although it's using about 3 times the bandwidth of 24-bit stereo audio. That's (a)
Of course, processing touches is the next step after that, and you have the client source as a guide for that (b).
yeah, I plan to do the usb hosting, partly as once i figure out how to do iso for the SP, I can also then add the eigenharp, which has more complex surrounding code, as it needs to upload firmware etc (though no complexity with tracking etc)... so SP is a good start for me.
as for touch, yeah, my current 'assumption' is i will dedicate one board (F4 or F7) to USB handling/touch, and then initially spit out midi (including mpe) data, but Axoloti will also later have support an inter-board protocol so we can connect other boards in a ring (hardware support is there, just needs firmware support added)
(of course the low level code will be compatible with any STM32F4/7 chipset, not just axoloti)
so initially Axoloti will be a standalone midi bridge... (it has usb midi host and device, and midi din support) ... some people are also working on a CV interface which they have connected to a modular, so once thats finished we will get that 'for free' :)
see http://community.axoloti.com/t/euxoloti-for-axoloti-eurorack-module/100
if you checkout the forum you will see, we are seeing a lot of excitement / involvement.. and thats with I think about 60% of boards delivered, and many still 'finding their feed' with the platform - so looking promising.
(note: Im just an early contributor and a 'fan' of axoloti, no financial connection etc. but do think its got an exciting future :))
I'll keep you posted.
There's also a Linux port in the works using libusb
is it possible to get a look at the 'work in progress' of this, I don't care if its working or not, its just I'm more familiar with libusb than the OS X api, so this would help me figure out the STM implementation.
(either send me a copy, or just chuck it in a branch in github and I can take a look from there)
These are at:
git@bitbucket.org:per_eckerdal/soundplane.git
git@bitbucket.org:per_eckerdal/madronalib.git
a few small madronalib changes were needed for Per's Linux port. I have yet to merge them into the master. When I do, the changes will show up in master on the madronalabs repo.
these are private repos, so no access ... no issue, I think I got the gist of whats going on in the Mac OSX versions.
Just an oversight, not intended to be private.
I merged Per's versions with the master branch today and tested everything on Mac OS. With success! I have no way to try the Linux version but I should have a Linux computer here soon. I'm sure you will beat me to it! If Axoloti is running something Debian-like, this should be straightforward.
Note that this moves us to a cmake-based build—please see the build instructions.
EDIT: cool, have this version running on both Mac OS X and Ubuntu :)
btw, but you have issues with your merge...
you have created a git submodule dependancy to bitbucket which is private...
I changed this to point to your github repo in ./gitmodules.
linux, yeah I have 32 and 64 bit versions of debian and ubuntu here, so I can test these, I can also test it on a PI2 which i run a debian derivative on.
axoloti, is not unix based... it uses a real time OS, as do all micro-controllers, this actually is a real advantage as it means you get much more reliable timing :)
cmake... cool, no problem familiar with this.
(Ubuntu has some issues with performance/packet counts, but Im sure you know this, and its work in progress... but looking good)
Good to hear, thanks for the info. If you come up with anything else of course I'm interested. I have a BeagleBone Black arriving here soon to try out.
ok, some success, and some failure on the PI2 :)
first I got it all compiling, main bit was figuring a way to make MadronaLib compile, as I needed to go from SSE to NEON fpu instructions (Intel vs ARM) . after that all proceeded pretty smoothly.
results:
UI is very sluggish, unsurprising, I have the same on EigenD... basically for these machines you will want a headless setup.
(Its not helped by the fact that I use remote X, probably would be better if I connected via HDMI, but still, not what you want.Soundplane is detected, and connected to... I get power and serial numbers but no pressure data, it looks like its all zeros, but without errors. this was also confirmed using HelloSoundplane
Ive not double checked this, but I fear this is going to be the same issue Ive hit with the Eigenharp Pico.... basically there appears to be a bug in the PI usb kernel module, that means its not waiting for data correctly on iso traffic.
i raised an issue on the kernel here..
https://github.com/raspberrypi/linux/issues/903
odd though, I thought the issue was 1.1 vs 2.0 devices, but the Eigenharp Alpha (usb 2.0) works fine, and the soundplane is a 2.0 device. (the Pico which didnt work is 1.1)
hmmmmm.... food for thought.
Hello! I've been away for quite a while, haven't I? Looks like I'm not the only one hacking away.
In case you guys are interested, after I began looking into support for the soundplane, I was also adding support to the 'b for a few other USB devices like USB MIDI, monome, etc. The arduino-based code was getting cumbersome and I was stretching the little SAM3X to its limits. I was also spending too much time writing USB drivers from scratch.
So I started looking for unix-based systems that supported DSP - as I was very interested in audio as well. The new SHARC+ chips really interested me - dual SHARC cores and an ARM core with buildroot. I wrote quite a bit of DSP code for that core - some spectral manipulation and an impulse response reverb.
Perhaps luckily, I was unable to get access to the early developer program, so could not get their linux kernel. So I started looking around, and around then the Beagleboard x15 was announced. It's got a TI AM5728 - dual 1.5GHz ARM15 cores, dual C66x DSP cores, lots of RAM, etc.
The x15 doesn't quite fit the eurorack format, but the processor is perfect and there are other modules available that are a better fit. I've been working with that for a couple of months now and love it.
I've since gotten USB MIDI support, monome support, and gamepad support running and now it is time to tackle the soundplane! I've spent only a few days so far porting some code, so I don't have much progress to report.
I can say that the device is listed in lsusb, but when running my simplified hellosoundplane, the LED does not come on and no frames are received.
the new b2/dsp will support HDMI video for hackers, but everything I'm building is headless, so after I get past getting some USB frames (hopefully with more luck than the RPi), then I'll be looking at migrating only the core madronalib to NEON - and perhaps with a little C66x help?
I'll let you know how it goes.
Just as a sanity check - Can I assume that this entry in lsusb indicates everything is fine?
Bus 003 Device 057: ID 0451:5100 Texas Instruments, Inc.
Couldn't open device, some information will be missing
Device Descriptor:
bLength 18
bDescriptorType 1
bcdUSB 2.00
bDeviceClass 255 Vendor Specific Class
bDeviceSubClass 0
bDeviceProtocol 0
bMaxPacketSize0 64
idVendor 0x0451 Texas Instruments, Inc.
idProduct 0x5100
bcdDevice 0.95
iManufacturer 1
iProduct 2
iSerial 3
bNumConfigurations 1
Configuration Descriptor:
bLength 9
bDescriptorType 2
wTotalLength 41
bNumInterfaces 1
bConfigurationValue 1
iConfiguration 4
bmAttributes 0x80
(Bus Powered)
MaxPower 230mA
Interface Descriptor:
bLength 9
bDescriptorType 4
bInterfaceNumber 0
bAlternateSetting 0
bNumEndpoints 0
bInterfaceClass 255 Vendor Specific Class
bInterfaceSubClass 0
bInterfaceProtocol 0
iInterface 4
Interface Descriptor:
bLength 9
bDescriptorType 4
bInterfaceNumber 0
bAlternateSetting 1
bNumEndpoints 2
bInterfaceClass 255 Vendor Specific Class
bInterfaceSubClass 1
bInterfaceProtocol 0
iInterface 5
Endpoint Descriptor:
bLength 7
bDescriptorType 5
bEndpointAddress 0x81 EP 1 IN
bmAttributes 37
Transfer Type Isochronous
Synch Type Asynchronous
Usage Type Implicit feedback Data
wMaxPacketSize 0x0184 1x 388 bytes
bInterval 1
Endpoint Descriptor:
bLength 7
bDescriptorType 5
bEndpointAddress 0x82 EP 2 IN
bmAttributes 37
Transfer Type Isochronous
Synch Type Asynchronous
Usage Type Implicit feedback Data
wMaxPacketSize 0x0184 1x 388 bytes
bInterval 1
Interestingly the syslog does show the correct manufacturer and product even though it's not in lsusb, so I feel a little better.
libusb is still reporting status 0 in hellosoundplane, but... never knew serial number 6. Is that right?
Mar 15 11:44:02 nw2s-0001 kernel: [463066.642188] usb 3-1.1.2: Product: Soundplane Model A
Mar 15 11:44:02 nw2s-0001 kernel: [463066.647854] usb 3-1.1.2: Manufacturer: Madrona Labs
Mar 15 11:44:02 nw2s-0001 kernel: [463066.653476] usb 3-1.1.2: SerialNumber: 0006
Hi Scott
Ive actually got the soundplane reporting pressure data (etc) on a Beaglebone Black.
though it doesn't at the moment quite look correct, as the touch tracker is not quite right.
Im not sure if this is my SSE to NEON layer, or other code Ive written and how its configures the soundplane/touch tracker, and haven't had time to check
I did this quite a while back now, but then got sidetracked on other projects, but Im hoping to get back to it soon. and when I do I'll have a new 'OSC' layer which will allow me I hope to debug the issues.
its tantalisingly close :)
(oh and the pico which didn't work on the RPi does work fine on the BBB, that bit of the code Ive got up and running)
Nice - is it available on a repo anywhere? I'd love to give it a shot and see if I can help at all.
Now I'm in business. I was missing the udev rule. Getting frames now! (And yes, they have actual values in them, so I don't have the RPi problem)
SUBSYSTEM=="usb", ATTRS{idVendor}=="0451", ATTRS{idProduct}=="5100", MODE="666"
Moving on to peak detection!
Hiya, I am reading about this exciting work with great interest and would love to be helping out better were it not for crunch time on Virta!
The endpoints above look good, but I am not really fresh on this USB stuff.
Version 2 of the touch tracker will be coming fairly soon after this plugin release. It should be clearer and faster and better suited to porting.
Looking good! Got this running on the AM5728 with no optimizations at all. I'll probably spend some time cleaning up a bit and seeing if I can optimize some of the vector code with NEON at all before tackling the tracking work.
| . : |
| .;x: . . . . . :: |
| ...=X@@X... ;=X@@$+. |
| :+x@@@@@x;::. . .. ....+$@@@@x: |
| ...::+=@@@@@=;.::. .. .+$@@@@$+... |
| .;:;XXX+. . .::++:: |
| .::=; .:;;.. |
| :. |
signal @ 0xbec86b38 64x8x1 [512 frames] : sum 61.9141
(the ampersands didn't translate - converted to @)
@scott...
I don't have a repo with the code in, but if you wish to collaborate, then you could email me.
( mharris AT technobear DOT com)
Ive got what you have so far, and Ive got it running on Linux (x86_64, PI, BBB) and Mac.
Ive also 'ported' the ML DSP and parts of the MLApp and TouchTracker.
(Ive used SSE2NEON as a starting point rather than updating the code, as at this stage I think its important to remain compatible with the ML code base!)
I made some minor additions to SSE2NEON
https://github.com/jratcliff63367/sse2neon/blob/master/SSE2NEON.h
Ive also got a new version of the hellosoundplane, (touchtrackertest)that integrates the touch tracker.
I re-tested this yesterday, and found that this touchtrackertest has the same behaviour on the mac as BBB, i.e. its to do with configuring the touch tracker rather than the SSE2NEON code... which his 'good news'... it also means I can debug it in Xcode :)
current status, touch trackers is reporting fake touches..
Im guess this is probably something to do with the calibration or normalisation maps.
(the other more static values Ive set as they are in the app)
Ive two ideas how to move this forward.
debug the fake touches, using the current setup
(and compare to how this goes in the soundplane app)load the calibration/normalisation data from the soundplane json file.
(id been avoid this for the test to simplify things, but mid-term i need it, as I don't plan to have calibration in this app... rather Id used the calibration thats made in the soundplane app on my mac)
thoughts?
@Randy, I know your busy with Virta, but would you perhaps have a few minutes to look over my test app, and see if Ive got anything obvious missing when initialising the touch tracker?
... its good to know others are interesting/looking into this, I think we can crack it easily together.
Mark
It looks like you are doing great, you are over the biggest hurdle!
I know I would end up spending the whole day on it if I switched gears to look at your code, so I'm going to have to wait to join this party until Virta is released.
Thanks! I'll reach out to via PM and we'll coordinate. I'm still wrapping my head around the code, but I certainly got a lot further than I thought I would in just a couple of day's hacking.
Before I get there, I'm going to POC moving some of the sse code to neon by hand rather than through a wrapper. The pressure decoding is hitting about 2.5% CPU with no optimizations, so I should be able to get that down a little and it will help me confirm how I want to start writing the optimized C code. (Probably #ifdef ARM_NEON)
...well, that's exciting. Just optimizing the few sse cals to neon changes the cpu utilization for pressure decoding from 2.5% to 0% as measured wtih pidstat.
I was looking through MLSignal and MLDSP and see a lot of vector operations that aren't optimized yet. Makes migration a lot easier for sure, but also some room for improvement as I get a little further. Back to other commitments for a while, but I'll be picking back up looking through the tracking code. Thanks for the help so far!
s
Very cool! I'll be excited take a look when I'm back to Soundplane work.
When you do pick up the tracking code, I would encourage you to look at the new-tracker branch. It's not fully working, so that's a disadvantage, but it is on the way to a much better solution than the old tracker, and maybe far enough along that it will be useful.