ts3n3r

[ts3n3r]

vCloud director header size errors

vCD would throw header too large errors because of poor cookie handling. While you can strip headers and play with the at the LB level, there is the option to just increase size at the jetty level:

-Dorg.eclipse.jetty.server.Request.maxFormContentSize=500000

This should be added to your vmware-vcd-cell-common script on your appliance/instance.

[ts3n3r]

vCloud Director 10.2 and external Postgres DB

If you are not using the appliance for vCD for any reason, and you are sticking to the supported CentOS  distribution  for your cell - there is one DB change required  on the DB server side after upgrading to 10.2. IMHO this could occur in earlier versions, as well. I have not tested  it.

Having a relatively large vcd  setup that services few thousand VMs now, our cell has production load. It seems though that DB connection pooling is not really well handled in the application. Thus,  one could get  connection pool exhaustion. Errors like these in the UI are one of the indications: [ 2c6d3e1e-8fa5-4f4f-859b-96781df7507b ] Cannot open connection - [pool-jetty-12388] Timeout: Pool empty. Unable to fetch a connection in 20 seconds, none available[size:75; busy:75; idle:0; lastwait:20000] One could easily check that is indeed the case: [root@vcd2 ~]# netstat -punta | grep :5432 | wc -l 76 Now, it turns out the vCD appliance also introduces some tune-ups on the postgres DB - namely setting an appropriate value for idle_in_transaction_session_timeout  By default on Centos postgres 10 package from the official postgres rpm repo, this is set to 0 - which means  there is no timeout for long-running transactions. Best practice would  be to handle these in the app. However, vCD  requires a value of 1 hour in the config.  Either apply it run-tme: ALTER DATABASE vcloud SET idle_in_transaction_session_timeout = 3600000; or in postgresql.config file: idle_in_transaction_session_timeout = 3600000   # in milliseconds, 0 is disabled

#postgres #VCD #vcloud #vmware

[ts3n3r]

Get me a pint of beer

As an attempt  to get to blog more frequently, here is a a donation link using Ko-Fi: https://ko-fi.com/tsener

[ts3n3r]

VMware vRO 8 SNMP  traps

//HOWTO

Set the SNMP Trap Port

Before setting the SNMP Trap port via the workflow you have to open that port in the firewall and forward the traffic on that port to vRO podLogin to the console on the appliance and run the following commands

# Create a variable for the SNMP port (81-32767) SNMP_PORT=22222# Create a kubernetes NodePort service for that SNMP port over UDP kubectl -n prelude expose deployment vco-app --type=NodePort --name=vro-snmp-$SNMP_PORT --port=$SNMP_PORT --target-port=$SNMP_PORT --protocol=UDP # Add the port as nodePort to the service kubectl -n prelude patch svc vro-snmp-$SNMP_PORT --type='json' -p '[{"op":"replace","path":"/spec/ports/0/nodePort","value":'$SNMP_PORT'}]'# Patch the service so the source IP address can reach the vRO pod kubectl -n prelude patch svc vro-snmp-$SNMP_PORT -p '{"spec":{"externalTrafficPolicy":"Local"}}'# Create firewall rules for that port over UDP iptables -A INPUT -p udp -m udp --dport $SNMP_PORT -j ACCEPT

Known issues

- After an upgrade the service needs to be recreated by running the commands again

- Only ports 81-32767 can be used

#vro #snmp #vmware

[ts3n3r]

Ansible for air-gapped VMs

Configuration of fresh-installed Linux systems, no matter the distro, before having network attached.. Given that a better approach would be some temporary management network, If you do not want to or simply cannot  manage for such an option in the moment, you can ship Ansible just fine in a chroot. I was looking for a way to pack up Ansible  either as an rpm or a static linked binary; Turns out that is not that easy.. Finally I come up with  scraping  the contents of a docker image with ubuntu/alpine and Ansible,  tar-in it and running that in chroot. Of course, one would need an SSH server running on the hsot as well, so  the  chrooted Ansible would connect to the host outside the chroot, sort of elevate to manage the host.. the Ubuntu tar with Ansible was about 90 MB which can be injected as a file, or base64 encoded and dragged upon via vRO  or another tool that speaks with VMware tools. Placing the whole thing in /tmp  or clean it up after execution would  do to keep the image pristine with just the changes,  and no management overhead or attack surfaces exposed later. The chroot could be omitted by using something more light-weight like cdist, but  one would not benefit  as much as from Ansible’s large collection of ready-made playbooks.

#ansible #vmware

[ts3n3r]

VCD-CLI

VMware has  released a python pyvmomi based  CLI for managing vCloud DIrector 9.X As it is a good candidate  for fire-and-forget automated tasks,  it would be great to have it as a docker container. I’ve prepared one based on Alpine Linux, and published it to DockerHub. Here is a link :

https://hub.docker.com/r/pscoelab/vcd-cli-alpine/

#vcd #vmware #cloud #docker #python

[ts3n3r]

NGINX  in front of syslog GELF

More notably, problem is how would you load-balance  UDP traffic. I have a Graylog2 cluster to be storing a lot of logs for  few hundred  ESX/AD/VCSA/Other  virtual Apps.  Searching around for a solution, three obvious options appear.

The fist option would be DNS round-robin and iptables/ufw.  This is not very robust imho, albeit it  should be most-performant in theory. DNS round-robin is just won’t do for anyone who wants to have control. Firewall worked of course,  I already had this setup  without the DNS round-robin for the single-node tests.

Second option is Keepalived -  I already use it, but never as a load balancer, but rather it’s uCARP (VRRP) capabilities for  NGINX nodes. config  syntax is somewhat old-school there, but one  would not mind, looks like a robust piece of software.

The last option, which I also turned out  to prefer, is NGINX 1.2.x  with streams module. NGINX is a great piece of software, with awful documentation.. getting  a packaged version with the required modules turned out to be difficult.  There are some discussions about it’s caveats after you have it installed... Fortunately for me, just before  deciding to compile from source and wast some more time, I decided to look for something ready-made at dockerhub. There is a non-official image there you can fork off, and get  a really nice container with UDP NGINX proxy, with separate config files for the TCP/web part and stream/UDP.

Additional config was required  to get my UDP GELF messages to get to their final destination:

ethtool --offload  ens192  rx off  tx off ethtool --offload  ens160  rx off  tx off

Otherwise, UDP checksum is no-go, alas this is for Virtual machines only.

In re-using we trust.

#nginx #vrrp #gelf #syslog #container #udp

[ts3n3r]

FOSDEM 2017

I attended FOSDEM 2017 in Brussels this year, with some friends. It was quite refreshing experience  seeing the  great ideas and some of the struggles with the open-source community, specially in regards to  virtualization. Some  quite cool ideas  on the networking side,   lots of programming workshops, which I did not attend   in favor of the  more infrastructure  related talks and  quick lightning talks  for small and very cool ideas.

[ts3n3r]

VXLAN packet drop check

mpssh -f iplist.lst -v "net-vdl2 -S -s dvSwitch | grep tx.drop.invalidState" | grep tx.drop | cut -d: -f2 | grep '[0-9]' | awk '{print $1}' | paste -s -d+ |bc I am using mpssh for this check, if the output differs from 0 you oughta check your dvswitch teaming settings ..  Good for nagios or snmp extend.

#vxlan

[ts3n3r]

pw(n)d

bash-4.3# cd // bash-4.3# pwd // bash-4.3# cd /// bash-4.3# pwd /

fine if you expand it from within the

shell, but what if you are using pwd  to pass current path to another program? 

[ts3n3r]

Cam hacking

Been awhile since my last post, as no time resource is available for ballooning  all of the stuff that I find interesting. I got myself a security camera on the cheap - a TP-LINK one, which the cool guys at IOactive   fancy as well... Here is how to root it and squeeze a little more   out of it: http://blog.ioactive.com/2016/03/got-15-minutes-to-kill-why-not-root.html

#tp-link #camera #root

[ts3n3r]

Your first, your last, your everything sshpass command on an ESXi

for HOST in `cat /tmp/myhosts.lst`; do sshpass  ssh-copy-id  $HOST ; sshpass ssh -o StrictHostKeyChecking=no root@$HOST "cat /.ssh/authorized_keys >> /etc/ssh/keys-root/authorized_keys ; rm -f /.ssh/authorized_keys ; ln -s /etc/ssh/keys-root/authorized_keys /.ssh/authorized_keys"; done

There. Now go and use f*cking ssh keys.

[ts3n3r]

SUDO-MUDO

I will just paste this here:

root@sanctuary:/opt/etherpad-lite/bin# su - etherpad No directory, logging in with HOME=/ Cannot execute /bin/zsh: No such file or directory *** glibc detected *** su: free(): invalid pointer: 0x00000000006084c0 *** ======= Backtrace: ========= /lib/x86_64-linux-gnu/libc.so.6(+0x76a16)[0x7fdab3822a16] /lib/x86_64-linux-gnu/libc.so.6(cfree+0x6c)[0x7fdab38277bc] su[0x40625a] su[0x402e4b] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xfd)[0x7fdab37caead] su[0x403611]

#sudo #memoryleak

[ts3n3r]

OVF Y  U NO STANDARTIZE?

“allocation unit MegaBytes does not match the rules for CIM programmatic unit specification used in OVF descriptors.”

Nice.. well, edit your .ovf file and  change 

<rasd:AllocationUnits>MegaBytes</rasd:AllocationUnits>

to

<rasd:AllocationUnits>byte2^20</rasd:AllocationUnits>

Seems common when  importing  in vCloud from a Virtual Box  OVF.

endOfStory

[ts3n3r]

vCloud Director storage over-commit

Have vCloud Director in a production lab for some time now,  and would like to share some experience regarding thin provisioning under it.  VCD is a product sitting on top of vCenter. While vCenter has resource pools  and thin provisioning working great, VCD add another layer  for organization quotas. When you are creating such an organization, you can specify the resources ‘reserved’ for this organization. There is also an option to use thin provisioning on a global scale for your organization. There is the pay-as-you-go option for provisioning that seems most appropriate for overcommitting purposes, as it doe snot have any hard limit quotas set.  However, if you choose allocation or reservation models, and do not overcommit your organization storage quota, you would not be able to benefit from your disks being thin. In short, you need to present e.g. your 10TB LUN as  300% storage utilization quota, if  you kinda expect each VM to use three times less storage than provisioned. That is, if you have any idea of the usage that you would get - which could be rather hard in  many situations. This practically kills the whole idea of having storage quota at all. In order to mitigate this, one should provision for e.g. 500%  of the LUN size, and use other tools to monitor the real utilization. Bottomline: for thin provisioning and overcommitting use Pay-As-You-Go model... 

[ts3n3r]

Github, you need MOAR form sanitizing..<br>

IMHO it's a nice ad space ;) This goes into a report issue body description:

<meta name="description" content="Lorem ipsum dolor sit amet, consectetur adipiscing elit. Ut eros lectus, dignissim a dolor quis, varius aliquet lorem. Aliquam erat volutpat. Quisque sagittis nulla vel quam auctor, vel pharetra nibh dapibus. Nunc id nulla vel mi dignissim sagittis ac et eros. Fusce aliquet ullamcorper ligula. Integer efficitur consectetur cursus. Nullam elit augue, aliquet dignissim cursus ut, volutpat ac dui.">

[ Y U NO XSS? ]  It does escape the special chars on input.

But aye, you can try to fool some auld browsers with 

[Meow] (javascript:alert('XSS'); )

More on that here.

Still, your text is saved there waiting for crawlers and alike. 

P.S. I could not resist... ; ) That's some issue reported on Ruby-On-Rails's github page.

#github #hacking

[ts3n3r]

Two-factor auth on Android with NFC Ring

Got the NFC ring first batch, a really nice idea to put NFC chip (two of them actually) into a wearable titanium ring. There are two Google Play apps that you can download along with your ring to make use of it;  NFC Ring Control is the app  which you can use to read or write arbitrary info and setup your ring; The other one is NFC Ring unlock app - which basically  loads a second lock screen after you unlock your android PIN or pattern. At first glance, the lock screen seems functional; If you somehow manage to loose your ring, there is an option to unlock the device with your google account. Which would be perfectly OK by the way, BUT:

The account password is not even masked when you type it ? It's a text instead of password field 

The login does not work with  Google two-factor enabled

So basically, if I have Google's two-factor enabled and I loose my ring, I would have to reset to factory settings in order to have my phone usable. There are setting of course which set behavior upon phone calls etc. so I'd still be able to answer phone, enter android's settings menu, turn off NFC,  enable debug ( hmm  ;) ) etc. etc.

Other than these two security issues, there are other concerns as well:

Google hangouts does not really behave as phone app, so it does not get the unlock exception during hangouts video call; I can guess same goes for Viber, Skype, and their kind. I am also not able to unlock the phone at all during a hangouts call/ring - I'd have to debug to find out why. 

Same issue is valid to alarm applications - I am not able to  stop my vanilla android morning alarm when NFC lock is in place. It just does not read the rind for some reason when the alarm is active.

Still, it is a smart gadget, and somewhat restored my faith in crowd funding and Kickstarter after previous unpleasant experience. 

As far as it goes as a phone lock mechanism, the ring  is good only for buying time for you to issue remote wipe of the device when lost/stolen. I would not trust it further than that. Now days  PIN and pattern are just not enough  - they are easily bruteforce-able.

Link: http://nfcring.com/

#security #nfcring #nfc