details on how we managed to recover an esxi node

warning // lessons learned

the newer esxi systems don’t live on a normal file-system. Those are embedded in an sqlite3 store.

I had to find out what exactly this command does (or the equivalent in the UI, enabling TSM-SSH service policy)

    vim-cmd hostsvc/enable_ssh

that helped identify which files are getting modified

    find etc -mmin -3

==>

    etc/vmware/configstore/current-store-1
    etc/vmware/hostd/hostsvc.xml
    etc/vmware/.backup.counter
    etc/vmware/dvsdata.db
    etc/chkconfig.db

so I had to use another kind of trickery

    vi etc/vmware/hostd/hostsvc.xml

    <ConfigRoot>
      <mode>normal</mode>
      <service>
        <TSM>on</TSM>
        <TSM-SSH>on</TSM-SSH>
      </service>
    </ConfigRoot>

    vi etc/chkconfig.db

    /etc/init.d/SSH on
    /etc/init.d/ESXShell    on

on workstation

cat ~/.ssh/id_rsa_sample.pub

on esxi node

    /usr/lib/vmware/sqlite/bin/sqlite3 \
            var/lib/vmware/configstore/backup/current-store-1

    select * from config where Name = 'ssh_auth_keys';

    insert into config(
        Component,
        ConfigGroup,
        Name,
        Identifier,
        Version,
        UserValue
    ) values (
        'esx',
        'system',
        'ssh_auth_keys',
        'root',
        '2.2',
        '{"username":"root","auth_keys":["ssh-rsa YOUR-OPENSSH-FORMAT-RSA-PUBKEY comment"]}'
    );

ready to go & acceptance

check that TSM-SSH is listening

    ping real-esxi
    watch nmap -p 22 real-esxi

and attempt to login

    ssh real-esxi -l root -i ~/.ssh/id_rsa_sample

troubleshooting

esxi kex_exchange_identification: Connection closed by remote host

resources

https://stackoverflow.com/questions/67000681/kex-exchange-identification-connection-closed-by-remote-host

https://serverfault.com/questions/1015547/what-causes-ssh-error-kex-exchange-identification-connection-closed-by-remote

https://stackoverflow.com/questions/69394001/how-can-i-fix-kex-exchange-identification-read-connection-reset-by-peer


https://serverfault.com/questions/547121/persisting-esxi-bootbank-boot-cfg-across-reboots

FW https://williamlam.com/2022/12/esxi-advanced-kernel-settings-reference.html

https://github.com/lamw/esxi-advanced-and-kernel-settings


https://www.baeldung.com/linux/recently-changed-files

HOME | GUIDES | PLAYBOOKS | LECTURES | LAB | CONTACT | HTML | CSS
Licensed as MIT