Wednesday, May 13, 2009

Project Roadmap

The project roadmap is:

Phase 1:


Add a different switch for destination port in sftp (-d)

If in the future we eliminate scp and instead create a softlink to sftp named "scp", we can implement old/new behaviour for -P switch according to the launched program name.

Implement -2 -4 -6 -c -q -i switches by passing them directly to ssh(1).


Phase 2:


Implement the -r switch, to implement recursive behaviour. This operation will be optimized by pipelining in a later phase.


Phase 3:


Implement -p switch, to preserve original file times. This switch will cause sftp to set/get the appropriate attributes in SSH_FXP_OPEN messages, or if needed, with a separate message.


Phase 4:


Implement -l switch, to limit bandwidth. Research the possibility of using scp's bwlimit().


Phase 5:


Review, improve if needed to close final bugs or implement any needed functionality, and merge tab completion patch with support for command completion, local file completion and optionally remote file completion, if the OpenSSH's dev team agrees on the remote functionality inclusion.


Phase 6:


Improve the user experience in the interactive client, such as allow a put/get with multiple files specified as parameters. Research for improvements for other commands, taking suggestions from the community and also getting inspiration from lftp client functionality.


Phase 7:


Improve sftp-server and/or sftp client so it can work on paths where some directories are traverse only, ie, not searchable, and design a solution which ideally will avoid extra round-trips.


Phase 8:


Implement the pipelining of readdir() calls, and pipelining of open/read/writes with a small window for file handles in sftp multi-file transfers to avoid unnecessary delays in multi small file transfers.


Test thoroughly for any regressions, and add new regression tests if necessary.


Phase 9:


Implement auto-tuning of best settings in each connection for –B buffer_size and –R num_requests of sftp client in order to get the best speed out of the network link. I will experiment with different values to understand how sftp will benefit the most in different network environments, and design an algorithm to automatically manage this settings.

I can make sftp use auto-tuning by default, or make it a command line parameter for the user to enable it on-demand, and document it on the man page.

Test this functionality extensively, ensure we don't introduce any security bug and add new regression tests if necessary.


Phase 10:


Research and implement support for resuming file transfers, using djm's existing patch for downloads.


Phase 11:


Write the documentation to all the previous work, and create regression tests for sftp's new funcitionalities and test it for compatibility with scp


If there's time left, close as much relevant bugzilla items as possible, such as:

Bug #430 - Could add option to sftp-server to disable write access

Bug #831 - Allow agent forwarding in sftp & scp

Tuesday, May 12, 2009

Phase 1 completed

I'm happy to report the phase 1 has been completed and the patch submitted by e-mail. My Powerbook is working really good now with the full GNOME environment from ports-stable, and -current kernel, userland and xenocara, and everything seems to work well, even the right button click tho there's no DRI in PowerPC. I had to apply the workaround to the kernel to avoid Ultra-DMA modes downgrade on my disk, and use a US layout on the keyboard to be able to access the programming characters under GNOME, so if anyone out there has a portuguese powerbook layout for X.org/Xenocara, i'd be happy to hear about it.

Cheers

Tuesday, May 5, 2009

Update on all the activity that happened so far

First status update will report on everything that i went through so far in my project.

I started to work with OpenSSH-portable version CVS tree under Mac OS X. But since my work will be on OpenBSD-current OpenSSH tree, I started the effort to install OpenBSD-current on my only computer, an Apple Powerbook G4 17" 1,67 Ghz.
I'm using OpenBSD macppc port, and due to a lack of PowerPC virtualization software , I decided to install the latest snapshot of OpenBSD natively on my Mac.
After all data was safely backed up, I managed to live resize my Mac OS X partition to make room for OpenBSD and installed it without any significant problems. It turned out later the disk space allocated (3 Gb for / and swap) wasn't enough for recompiling the whole system, xenocara, and install some ports, so i proceeded to downsize Mac OS X partition again, and reinstall OpenBSD.
I had one recurring problem since the beggining of running OpenBSD on my Powerbook, that is the constant kernel messages about wd0 disk timeouts, and mode downgrade, from Ultra-DMA mode 4 all the way down to PIO mode 4, exactly like described in this bug report March 2008 on OpenBSD-bugs ML. In PIO mode 4, any heavy access to the disk, like a CVS update, will cause an interrupt storm and the CPU will be 75% of the time handling interrupts, as shown by top.
This causes the whole system to run very slow, and after i updated kernel, userland and xenocara to -current, the bug remained, with either GENERIC kernel, or a slimmed down version. I also tried disabling bwi driver, reseting NVRAM in Open Firmware bootup prompt, but nothing helped.
In the meantime i also installed some packages available in the snapshots/ directory of mirrors, and compiled the remaining from ports -current , to run vim, cscope, and Gnome desktop, to be able to be more productive.
Unfortunately, under Gnome, the keyboard goes crazy and some keys fail to show up in the terminal, and every time i use Gnome (2.24) the keys seem to get 'stuck', like, after pressing one Enter/Return, Gnome continuously gets Enter/Returns, as If i'm still pressing the key. Any attempt at using the keyboard under Gnome is useless. xev shows key press/release normally tho.
This behavior rarely happens in FVWM, so that's what I've been using mostly to work on OpenSSH's code.
I had to fight some more days with Xmodmap to get key recognition of my Powerbook's portuguese keyboard and be able to type basic programming characters like {} and [] . I also spent some more time getting a useful xorg.conf.
So right now, i am going to try an hackish workaround for the Powerbook hard disk timeouts, until the issue is sorted in -current, and I will soon get a virtual machine on a computer running VMWare and install OpenBSD i386 to do most of my development there remotely with ssh.
I expect to submit the first patch really soon now.

Welcome!

Welcome to the official blog documenting my work on OpenSSH's sftp(1) improvement project, sponsored by Google under the Summer of Code 2009 program.

I hope to write status updates on a weekly basis, so everyone interested can closely follow my work developing this exciting project.

Cheers!