Summary: Mounting LDOM virtual disk files outside of the LDOM, or from the primary

Summary:

Needed a way to mount/access data from files stored on a files used as
an LDOM hard disk.  lofi looked like a possible candidate, but it does
not support files containing a partition table, therefore it cannot be
used.  It turns out iSCSI can be used for this.  A hint to use this was
provided by J. Oquendo's friend Travis.

Tim Bradshaw provided this useful link: http://www.docbert.org/Solaris/Jumpstart/Sol9/Sol9EA-iso.html which contains some useful details about the Solaris VTOC - [an alternate method I wanted to explore was looking at the partition table/VTOC and using dd to chop up the LDOM storage file into individual files, one per slice, then use lofiadm on each slice file to mount them. ]

Steve Franks added that it's a good idea to use ZFS volumes for LDOM storage as doing so can allow the primary to mount the volumes far more easily.  This certainly is a good option to implement, sadly we hadn't used ZFS in this case due to performance concerns with Oracle (if anyone has recommendations regarding the use of ZFS and Oracle and how it compares vs UFS or raw disk, I'd love to hear it.)


Thanks to all those that replied.


IMHO, ideally, lofiadm should be upgraded in the future to allow it to recognize partition tables and handle those, then tricks like this won't be needed.



Method to make this work (note this is hackish, use at your own risk,
#include <std-disclaimer.h> do not do this on production systems, YMMV,
etc. )
Read iSCSI Solaris documentation before following this.

1. Create an iSCSI target of the same size as the LDOM shared file
(you'll obviously need free disk space)

iscsitadm modify admin -d /path

Where path is somewhere that you have enough free space.  This tells the
iSCSI server where to share its block devices from.

#iscsitadm create target --size 36g target

(Obviously rename the words /path, 36g, target to match your environment)

In my case, I had a 36GB LUN on the LDOM, so built the same here.  I
believe this creates a sparse file as it was fairly quick, but it might not.

Once created look into the path.  It'll look something like this:

# cd /path
# cd iscsi/
# ls -Fla
total 10
drwxr-xr-x   3 root     root         512 Nov 13 11:29 ./
drwxr-xr-x   4 root     root         512 Nov 13 14:17 ../
-rw-------   1 root     sys          263 Nov 13 11:29 config.xml
drwxr-xr-x   2 root     sys          512 Nov 13 14:17
iqn.1986-03.com.sun:02:????????-????-????-????-????????????.target/
lrwxrwxrwx   1 root     sys           77 Nov 13 11:29 target ->
/path/iscsi/iqn.1986-03.com.sun:02:????????-????-????-????-????????????.target/
# cd target
# ls -Fla
total 75534390
drwxr-xr-x   2 root     sys          512 Nov 13 14:17 ./
drwxr-xr-x   3 root     root         512 Nov 13 11:29 ../
-rw-------   1 root     root     38654705664 Nov 17 14:23 lun.0
-rw-------   1 root     sys          347 Nov 13 12:07 params.0
# cat params.0
<params version='1.0'>
    <size>0x4800000</size>
    <status>online</status>
    <interleave>1</interleave>
    <bps>512</bps>
    <spt>576</spt>
    <cylinders>32768</cylinders>
    <heads>4</heads>
    <rpm>7200</rpm>
    <dtype>disk</dtype>
    <vid>SUN</vid>
    <pid>SOLARIS</pid>
    <guid>?????????????????????????????????????????</guid>
</params>

Note the values in the XML file.  I didn't have to change any of these,
but it's a good idea to know what's there incase you may need to.

2. Copy the LDOM file over the iSCSI partition's lun.0 file.  It's a
good idea to do a copy so you don't accidentally damage the original
LDOM file.


3. Disable tcp fusion to allow loopback iSCSI to work as some versions
of Solaris have a bug that prevents it from working,
run this if this affects you.  I experienced this issue on Solaris 10
2008.05, and this fixed it.

# echo 'do_tcp_fusion/W0' | mdb -wk


4. Map the drive over iSCSI.  Either use discovery mode:

# iscsiadm add discovery-address 127.0.0.1:3260
# iscsiadm modify discovery --sendtargets enable

(Note that :3260 isn't required, it's the default port)

or static like this:

# iscsiadm add static-config
iqn.1986-03.com.sun:02:????????-????-????-????-????????????.target.disk,127.0.0.1

Next, use the "iscsiadm list target" to verify that you can see the
target.  If this fails, see step 3, or do other debugging.


5. Run this command to refresh the disk list:

# devfsadm -Cv -i iscsi


6. Run format, the iSCSI device should show up.  You can expect access
to be slow.  Should look something like this:

      14. c6t4d0 <SUN-DiskImage-36GB cyl 60291 alt 2 hd 1 sec 1252>
         
/iscsi/disk@0000iqn.1986-03.com.sun%3A02%3A????????-????-????-????-????????????.target.target0001,0

I'd suggest selecting the disk and doing partition print from within
format so you know what slices are there before you mount them.

If this is an OS disk, obviously slice 0 will be the root slice, and
then once you've mounted it, under say /mnt, you can look in
/mnt/etc/vfstab and mount the remaining slices in their appropriate dirs
and so on.




The original request:

Ray Arachelian wrote:
> Hi,
>
> We have several T5240s with LDOMs on them.
>
> The setup is that the Primary domain has a LUN mounted off an EMC SAN. 
> This LUN is then sliced using format and a file system is created using
> newfs.
> Then we create a large file using mkfile and assign this file to the
> virtual disk server to be used by a guest LDOM as a virtual disk.
>
> Once the guest LDOM is booted, it sees this file as a virtual hard
> drive, and during jumpstart this gets sliced and file systems are made
> as usual.
>
> Is there a way from the primary (whilst the guest LDOM is offline, and
> without creating another LDOM and giving it this LUN) to mount the
> partitions inside this file?
>
> I know that lofiadm can be used to mount a file in this way, however
> since the LDOM virtual hard drive file contains a partition table, it
> doesn't seem that lofiadm can handle this.
>
> Is there any other way to access the slices inside these types of files
> in an emergency?  (read only is fine)
>
> I imagine something can be done using dd by telling it to extra N number
> of blocks and skip S number of blocks, but how do I figure out what
> those values are and the sizes of the slices inside the Solaris
> partition table?    (I suppose I could also create a 2nd lun and dd the
> file out over the block device of that 2nd LUN..)  Is there any better
> way of acheiving this than using dd?
>
> I suppose I'm also making an assumption here that the LDOM virtual disk
> service is using these files as straight disks, but they might have some
> sort of header, do they?
> _______________________________________________
> sunmanagers mailing list
> sunmanagers@sunmanagers.org
> http://www.sunmanagers.org/mailman/listinfo/sunmanagers
_______________________________________________
sunmanagers mailing list
sunmanagers@sunmanagers.org
http://www.sunmanagers.org/mailman/listinfo/sunmanagers