System in OS X 10.11.4 El Capitan
$ cd $HOME
$ ls .
> Dropbox
> BTsync
Code which creates unsuccessful symlink
ln -s BTsync/ Dropbox/
Code which creates working symlink
ln -s /Users/masi/BTSync/ /Users/masi/Dropbox/
Evidence in size difference with files
$ cd $HOME
$ ln -s $HOME/BTsync/wfdb.zip $HOME/Dropbox/
$ ln -s BTsync/wfdb.zip Dropbox/wasi.zip
$ ls -la $HOME/Dropbox/
lrwxr-xr-x 1 masi staff 41 May 17 22:45 wfdb.zip -> BTsync/wfdb.zip
lrwxr-xr-x 1 masi staff 46 May 17 22:45 wasi.zip -> /Users/masi/Dropbox/wfdb.zip
Evidence with directories
$ ln -s /Users/masi/BTsync/cam\ local/ /Users/masi/Dropbox/
$ ln -s BTsync/cam\ local/ Dropbox/camLocal2
$ ls -la $HOME/Dropbox
lrwxr-xr-x 1 masi staff 37 May 17 22:52 camlocal -> /Users/masi/BTsync/cam local/
lrwxr-xr-x 1 masi staff 25 May 17 22:53 camLocal2 -> BTsync/cam local/
and visually same in Finder
which shows that the two commands are not the same in El Capitan.
Using absolute PATH only works.
Evidence with Relative PATH
$ cd $HOME
$ ln -s ./BitTorrentSync/cam\ local/ ./Dropbox/camlocal3
$ ls -la Dropbox/
lrwxr-xr-x 1 masi staff 27 May 17 22:58 camlocal3 -> ./BTsync/cam local/
and a new screenshot
Why OSX Symlink cannot Relative PATHs?
Best Answer
This isn't about relative paths. Relative paths work fine.
If what you're trying to do here is create a symlink from Dropbox to BTsync, the problem is that Dropbox already exists. This means the symlink is created inside the Dropbox folder, named BTSync. Delete Dropbox and try again.
If you really meant to create a symlink inside the Dropbox folder, then both these commands work identically.