The mysterious case of broken SSH client (“connection reset by peer”)

Update: from the info I've gathered, this is most probably a problem with some Cisco IDS/DPI is running on the ethernet equipment. Workaround is available in the content below, I still don't know what's the real solution here (Cisco equipment config? update Cisco firmware?)

---

Starting with 5.7p1, ssh client on specific environments fails connecting to specific (usually old versioned) servers. I reproduced it on a particular network, while trying to connect using new ssh client (5.8p1, Ubuntu 11.04) to an old server (default SSH server on RedHat 5.4).

Motivation

This issue is around for quite a while, but is very tricky to reproduce or understand. What bothered me most is that many people reported it to different forums, each posting only a few (different) pieces of the puzzle. So my motivation here is to try and summarize the relevant info from multiple places. I'll do my best to update this post when I hear something new.

Complete Fact list Continue reading

Perfect DHCP configuration for Solaris

I've already posted on configuring DHCP + Dynamic DNS for some operating systems.

I still claim that that's the only way to go for dynamic, trusted networks.

So after RedHat, Debian/Ubuntu, Windows, it's about time to stop the IP suffering with Solaris as well:

  1. Make sure /etc/nodename contains the FQDN, e.g. "snoopy.ac.uk"
  2. touch /etc/dhcp.<interface name> . There are two peculiar parameters I'm not fully sure about their meaning (how come they're not needed in other OSes?) - but here is what I used:
    1. add "primary" (a separate line) if it's the primary interface.
    2. Add "wait <seconds>" (a separate line) - I put "wait 60" to be on the safe side.
  3. /etc/hostname.<interface name> should have the hostname inside. NOTE! It's the hostname dhcp client would send (Option 12) to the server for Dynamic DNS registration. According to /etc/defaults/dhcpagent, the format is "inet name", e.g. "inet snoopy"

That should be it, then. Please add your comments and corrections.

Solution for Xsession + bash_completion problem

Update: just noticed that on a more advanced ~/.profile, I see it sourcing ~/.bashrc only if under bash (by using if [ -n "$BASH_VERSION" ]; ).

As Debian sid was just unfrozen due to Squeeze release, I recently get lots of package updates. I had the feeling that something is gonna break, even slightly, despite the relative high stability of Debian unstable.

And indeed, after 100+ updates, I could no longer login to GNOME. I'm not sure which package update caused this (bash_completion was updated to 1.3, yet the problem isn't necessarily there) . gdm nicely referred me to ~/.xsession-errors, which had something like:
sh: /etc/bash_completion.d/git: line 123: syntax error near unexpected token <'
sh: /etc/bash_completion.d/git: line 123:
done < <(git config -z --get-regexp '^(svn-remote\..*\.url|bash\.showupstream) 2>/dev/null | tr '\0\n' '\n ')'
sh: _grub_mkpasswd-pbkdf2': not a valid identifier
sh:
_grub_script-check': not a valid identifier

I couldn't find any related Debian bug report, nor hardly anything on the internet (just some Arch Linux bugs with no clear solution).

Eventually I've found out that /etc/gdm/Xsession uses /bin/sh (a crippled shell, even though it's a symlink to /bin/bash), which indeed fails with the same error when running 'source /etc/bash_completion'. This makes sense. bash_completion[*] should be bash-compatible, and may include bash-only non-standard-shell compliant syntax.

Wondering why Xsession gets to run ~/.bash_completion indirectly, I've found that I had the following two abominations:

  • 'source ~/.bashrc' command in ~/.xsession
  • 'source ~/.bashrc' command in ~/.profile

I'm not sure why I had these lines, but they shouldn't be there. Bash itself and only bash should source ~/.bashrc (and it's done automatically when bash starts). It makes no sense for other shells to source it.

So the solution was simply removing these lines from ~/.profie and ~/.xsession (actually got rid of ~/.xsession completely). Still, I'm not sure which update triggered this problem.

----

* Actually the problem was in /etc/bash_completion/git which is a part of the 'git' package and not the 'bash-completion' package. But all the same.

Configuring ctags for Python and Vim

Exuberant ctags is a cool, language-agnostic tool for creating tag files for your source code. Nice editors such as Vim, could use these tag files to implement the much needed 'jump to definition' feature.

Ctags is awesome, it supports Python, and is supported by Vim. It seems that the world is perfect and there's no reason to write a post about configuring it. Well... almost.

ctags has a little downside when using Python: it recognizes import lines as a definition, at least as of ctags v5.8. No need to explain why it's annoying in most cases. After 2 years of suffering, I've found it's possible to overcome this simply by adding the --python-kinds=-i option to the command line, or better: to ~/.ctags.

And just to make it complete, a quick cookbook style for setting everything up and using:

  1. Install ctags
    e.g. aptitude install exuberant-ctags
  2. Configure ctags.
    Add to ~/.ctags the following, one option per line:

    1. --python-kinds=-i
    2. optional: --exclude=<partial names of bad files/directories>. e.g. --exclude=*/build/* to exclude all files inside 'build/' directories
  3. Add a cron to rebuild tags, for instance:
    1 * * * * ctags -R -o ~/mytags ~/src
  4. Configure vim:
    add to ~/.vimrc: :set tags=~/mytags
  5. Use Vim:
    1. vim -t <tag name> to open vim straight on the tag
    2. Ctrl+] to jump to tag when over a word
    3. Ctrl+T to pop back
    4. :tselect or :stselect to open
    5. :tnext, :tprev to go to next/prev tag finding
    6. :help tags for more 🙂

Make DHCP auto-update the DNS

Update: added a new post on configuring Solaris, link below.

Motivation

In today's dynamic R&D network environments, it's not easy to keep the DNS records up-to-date: hosts are reinstalled/renamed/added frequently, virtual machines are so easy to deploy and destroy, DHCP allocates different IPs..

This even leads to pitiful situations, in which people get used to referring to computer by their IPs (or using /etc/hosts), because the DNS cannot be trusted to reflect the reality.

Solution

DHCP servers have the ability to send dynamic DNS updates, as they allocate IPs to clients. The great thing, is that it even works out-of-the-box on some operating systems. Still, if it doesn't, here are 3 things to care about, so to enable this feature: Continue reading

Booting Linux from iSCSI

What is this long post about?

iSCSI is standard for accessing block devices (e.g. disks) over network, just as if they were local SCSI devices. That's similar to AoE and FCoE, although the latter two are good for the LAN only, while iSCSI is over IP thus is good on WAN. This article would focus iSCSI but could be used as a base for doing similar things with AoE and FCoE.

So, iSCSI in the simplest configuration, allows us to mount and manage a data disk that is physically connected to a remote computer (the "server", aka target) from our own computer (client, aka initiator) .

On this post I'll discuss the deep details of the more advanced stage: having the root (also boot) disk on a remote computer, so client could remotely boot from it. Surprisingly it can be done even with relatively old hardware.

Continue reading

Surprising new keyboard shortcuts

(Well, surprising me, at least)

It's been a while since I wrote these two posts about keyboard shortcut, so I'll try to summarize some neat ones I've recently learned:

Unix Shell

  • Ctrl+U deletes all characters on the cursor's left. And.. keep it in the shell's clipboard.
  • Ctrl+K deletes all characters to the cursor's right (including the cursor's position), keeps in clipboard.
  • Ctrl+Y: paste clipboard.
  • Ctrl+R: search for string in history. Multiple Ctrl+R's search for the next matches in history. Btw, It's F7/F8 on windows command prompt.

Google vim keys and more

  • Google search: Google Experimental Search, enables vim keys for browsing the results! Super useful.
  • Google Calendar support vim keys (j, k, l, m) for navigating in the calendar. Type "?" for more keyboard shortcuts.

GNOME

  • Alt+F10 toggles between maximal window size and original window size.
  • Alt+F9 minimizes window
  • Alt+F8 switches to resize window mode
  • Alt+F1 opens the GNOME menu.

Thanks for Zohar and Aviv for presenting me with most of the above shortcuts.

DNS SRV records (or: why Google Talk doesn't connect through this network)

[ There is some blabla here about SRV records, See last paragraph for the direct solution ]

Recently, the bus home was replaced with a shiny new one, which also provides Wi-Fi access. That's brilliant - as long as I have a seat, I don't mind how long the ride home takes. I just hope it doesn't radiate too much =)

Anyway, I've noticed that I fail to connect to Google talk through my favorite client, Pidgin, when connected to this Bus' wireless LAN. It's really fun to do some debugging with xterm and wireshark with passengers watching with curiosity.

And to the point - I'm not sure where this standard is defined (XMPP?), but Google recommends setting the Google Talk "domain" as gmail.com. But where is the XMPP server ip/name taken from? It's not gmail.com, as the telnet gmail.com 5222 fails.

Quick sniffing tells that my pidgin (and probably other clients) use DNS SRV records. SRV records (RFC2782) are a pretty neat idea: it lets you query your domain for services. Why remember (or worse: guess) what's the name of the SMTP server? Instead, SRV records suggest a way to ask a domain what's the IP of the domain's major SMTP service.

The DNS name is of the following structure: _<service name>._<protocol>.domain.

In our case: _xmpp-client._tcp.gmail.com. Let's run this nice command line:

# host -t SRV _xmpp-client._tcp.gmail.com

_xmpp-client._tcp.gmail.com has SRV record 20 0 5222 talk2.l.google.com.
_xmpp-client._tcp.gmail.com has SRV record 20 0 5222 talk3.l.google.com.
_xmpp-client._tcp.gmail.com has SRV record 20 0 5222 talk4.l.google.com.
_xmpp-client._tcp.gmail.com has SRV record 5 0 5222 talk.l.google.com.
_xmpp-client._tcp.gmail.com has SRV record 20 0 5222 talk1.l.google.com.

So briefly, each SRV record holds host name, port number, priority and weight. A single service may have several records, thus providing some kind of high availability and load balancing.

SRV records are intensively used in Microsoft Domains since Windows 2000, and it's even possible that they invented it (they're signed in the RFC after some non-microsoft guy). In that case, kudos to Microsoft for either inventing or pushing forward a good technological standard. Haven't seen this frequently.

So, it seems that the DNS in this bus disrespects my SRV queries, and thus my Pidgin can't figure out who is the Google Talk server. Setting another DNS server solved the problem.