First run Shadowbox.init() : if you have static media elements on the page that use Shadowbox then allow init() to run setup(); if you don't then by all means set skipSetup:false - your decision.
If you subsequently load any shadowboxable media dynamically, eg. by ajax, then you need to tell shadowbox about it! You do this via Shadowbox.setup(), and the Advanced Usage section
of the documentation gives some pointers.
If you run Shadowbox.setup() with no arguments, it will search the current DOM for all relevant links and add new ones to its cache, or rebuild the cache entry of those it already knows about.
Alternatively, you can supply a collection of shadowboxable links as the first argument, with (optional) options as the second argument, eg Shadowbox.setup(links [, options]), where links can be a single element, an array of elements, or false/null if you simply want to supply a new set of options for current (and new) links found in the DOM.
So, for ajax calls, the 'oncomplete' (or equivalent) callback should include a call to Shadowbox.setup(), with or without arguments as you deem appropriate.
Note : Shadowbox.init() should be run at the first available opportunity on your page; re-running Shadowbox.init() at any later stage on the same page does absolutely nothing!
Combine literacy with curiosity and a whole world of information opens up to you
It doesn't matter which library you use - MooTools, jQuery, or YUI (that you're using) - the principle is still the same.