I've heard of Gigabit Ethernet. I just want to know what Ethernet cables are compatible for my Intel NUC DCCP847DYE.

Hello,

Quite new to "the world of hardware" and all option they have, but pretty used to tryout different technologies and concepts. Anyway, I found a i340-T4 on ebay pretty cheap and figured "hey, why dont I iSCSI boot my server - that sounds cool". So I bought the card after verifying that the auction page specified it supported iSCSI boot. I got the card but for some reason I only saw a PXE boot prompt. I read up a bit on the documentation and realized that I would likly need to flash the ROM of the NIC.

I downloaded the Intel Bootutil and tried to run it. It lists all the ports just fine, all 4 of them, when running the command without arguments. But when I run the command bootutilw32.exe -nic=2 -FE I get the error message:

"ERROR: Unsupported feature". So I googled a bit and found that apparently the NIC might only support combo images an hence the "enable" feature wouldnt be applicable since it was always enabled. So I ran the command bootutilw32.exe -nic=2 -up=combo I get the error message "Option ROM area in the flash is not supported for this device on port 2".

I googled some more and it turns out that maybe this might be an OEM card and hence would need the OEM vendors own tool. But how do I find out which the OEM vendor might be? Is there anything else I've missed when trying to get iSCSI working?

This is a standalone card, PCI-E 4x.

Please let me know if you have any suggestions... I have sent the ebay seller an email, but have not yet recieved a response. I have also tried with the HP tool I've found, but it still seems it is not working even tho the command ran without error messages when I used a config file as input.

Link to the card I bought on ebay (same NIC from same seller): New HP 593722 B21 NC365T Intel I340 T4 4 Port PCIe Gigabit NIC | eBay

Thanks in advance.

Hi Intel Folks I have a very strange Problem:

I got a Supermicro X9SRH-7TF Mainboard which comes with 2 Onboard 10GBe Ports through an X540-T2 Chipset. For about a year I had those two ports connected to a 1GBe Switch and everything was just fine. I got another Server which I recently added a 10GBe X540-T2 PCIe Card. I initially only connected the Servers physically without assigning IP Addresses within their Windows Server 2012 R2 OS. Then I was away for a week or something, came back and both 10 GBe Onboard Ports first showed up having trouble (Error Code 43) until after some trying to reapply the Intel PRO Driver Package (v18.7) they degraded to unknown Communication Device in the device Manager.

This is all I have from the Windows Event Logs: Log Name: System Source: Microsoft-Windows-UserPnp Date: - Pastebin.com

The Server with the PCIe 10GBe Card shows as connectivity "Disconnected" on the 10GBe Ports since the first server decided to not like those 2 Onboard Ports anymore.

This also is reproducable: In the meantime I've exchanged the Mainboard since I thought those onboard ports are screwed up but it's the same with the new Mainboard. I'm not sure where to look for this Problem (BIOS, Drivers, something else?) so I hope for your help asides my request towards SuperMicro.

Thanks

Hello all,

As it has been a couple of months that we started our MER internship here at Intel, I thought it would be a good idea to share our experiences of work and living. The aim of this topic is as well as our experiences help prospective MER interns, it would help current interns too. So this should be a gathering of useful information that fits to a "How to" question, such as "How to purchase a car?", or "How to get to work from your home?", or even "How to have fun here in Oregon?". So if you have such experiences and willing to share, please do a post to this topic.

Two months ago i built my new pc and i am using an i5 4670k. However i have some issues. It might be asrock but i have contacted them and am still awaiting an answer. I bought my cpu and paired it with a noctua nh-d14 for overclocking, however my cpu won't overclock at all! the only thing i can do is 4ghz at 1.9 volts for windows to not crash, but it gets way too hot! Also just 2 days ago all of a sudden my pc's boot and general file browsing slowed down by like 30%. I might need a replace but first id like to hear an opinion from intel themselves and wait for asrock to reply.

Hello,

We are working on Dell-T1650 PC running on Intel Xeon E3-1240 v2 and RHEL-5.1 is installed on PC.

Now, we have installed Intel's X-540-T2 NIC card on PCIe slot on same PC. It shares IRQ 16.

Due to Shared IRQ we can't get NIC's better performance in terms of lower latency on same PC. So, we are planning to assign separate or unique IRQs to NIC card.

How can we assign different IRQs to PCIe based NIC card (82599 Ethernet controller based X-540-T2)??

Cards in Slot1, 2, & 3 are all I350 cards.  There is a GUI / Powershell mismatch on whether or not VMQ is enabled.

I have enabled VMQ from the GUI via the device manager and by the Enable-NetAdapterVmq powershell command.  So in this instance, SLOT 1 Port 1, is the port I am referring to.  Enabling VMQ via powershell or by the GUI only affects the settings in the GUI.  Powershell still says enabled "False".  The check box in the Advanced settings / Virtualization settings unchecks and checks for Virtual Machine Queue within the GUI Properties of the NIC.  So the other thing is that SLOT 1 Port 3, a port on the same card says it is enabled in powershell.  Can anyone give me any insight into this issue and which setting I should believe?  I just installed 19.1 version of the intel proset.  I have a tendency to believe powershell over the gui.  Any insight would be helpful. Is there any service or anything I need to restart for powershell to show the change like the GUI does?

Last Question: is VMQ automatically enabled on a VMLB Team when used as an External Virtual Switch in Hyper-V 2012R2?  Note as you can see VMQ is disabled on all teamed NICs.

Reference Powershell output:

Thanks,

LCOE Tech

Hi,

Our topology is :

    Spirent <->Ubuntu 12.04 Server <-> our application  within VM where VF is directly attached.

on VF interface , created   vlan interface and spirent has also also vlan interface.

NIC : Intel 82599 ES

IXGBE Version : 3.15.1-k

IXGBEVF driver version :2.7.12

By default, VF is not receiving any packet , even ARP Packet.  After configuring vlan filtering from  VF, it gets packet  with stripped vlan tag.

There are couple of questions :

1. Is there any way to configure from VF, so PF doesn't filter any vlan , simply pass traffic to VF ? I checked Spec , it mentions about this and by default, it is enabled.

2. How to avoid stripping of vlan tag ? I  changed accrording to spec  but it  stopped receiving any vlan packet.

3. Even I tried to add all vlans(1 to 4094) , but it doesn't work. it looks like only fixed entries(most likely 8) can be added .

Please let me know how to fix this problem from VF side, can't change PF code.

Thx,

Atulya

Hello,

Quite new to "the world of hardware" and all option they have, but pretty used to tryout different technologies and concepts. Anyway, I found a i340-T4 on ebay pretty cheap and figured "hey, why dont I iSCSI boot my server - that sounds cool". So I bought the card after verifying that the auction page specified it supported iSCSI boot. I got the card but for some reason I only saw a PXE boot prompt. I read up a bit on the documentation and realized that I would likly need to flash the ROM of the NIC.

I downloaded the Intel Bootutil and tried to run it. It lists all the ports just fine, all 4 of them, when running the command without arguments. But when I run the command bootutilw32.exe -nic=2 -FE I get the error message:

"ERROR: Unsupported feature". So I googled a bit and found that apparently the NIC might only support combo images an hence the "enable" feature wouldnt be applicable since it was always enabled. So I ran the command bootutilw32.exe -nic=2 -up=combo I get the error message "Option ROM area in the flash is not supported for this device on port 2".

I googled some more and it turns out that maybe this might be an OEM card and hence would need the OEM vendors own tool. But how do I find out which the OEM vendor might be? Is there anything else I've missed when trying to get iSCSI working?

This is a standalone card, PCI-E 4x.

Please let me know if you have any suggestions... I have sent the ebay seller an email, but have not yet recieved a response. I have also tried with the HP tool I've found, but it still seems it is not working even tho the command ran without error messages when I used a config file as input.

Link to the card I bought on ebay (same NIC from same seller): New HP 593722 B21 NC365T Intel I340 T4 4 Port PCIe Gigabit NIC | eBay

Thanks in advance.

Hi Intel Folks I have a very strange Problem:

I got a Supermicro X9SRH-7TF Mainboard which comes with 2 Onboard 10GBe Ports through an X540-T2 Chipset. For about a year I had those two ports connected to a 1GBe Switch and everything was just fine. I got another Server which I recently added a 10GBe X540-T2 PCIe Card. I initially only connected the Servers physically without assigning IP Addresses within their Windows Server 2012 R2 OS. Then I was away for a week or something, came back and both 10 GBe Onboard Ports first showed up having trouble (Error Code 43) until after some trying to reapply the Intel PRO Driver Package (v18.7) they degraded to unknown Communication Device in the device Manager.

This is all I have from the Windows Event Logs: Log Name: System Source: Microsoft-Windows-UserPnp Date: - Pastebin.com

The Server with the PCIe 10GBe Card shows as connectivity "Disconnected" on the 10GBe Ports since the first server decided to not like those 2 Onboard Ports anymore.

This also is reproducable: In the meantime I've exchanged the Mainboard since I thought those onboard ports are screwed up but it's the same with the new Mainboard. I'm not sure where to look for this Problem (BIOS, Drivers, something else?) so I hope for your help asides my request towards SuperMicro.

Thanks

Hello to everybody

I have problem for installing Intel Proset Software that Enable Team & VLAN feature, but when i install the NIC Adapter Intel Pro 1000 MT in Windows Server 2008 R2, O.S. automatically will installed the drivers without problem but when i try to install the Intel Proset Software on its on the box CD, i see the Error:"cannot install drivers. no Intel  adapters are present in this computer" . I try to download & install the latest driver from Intel support web site Ver. 19.1, but there is the same problem !!! . So, please help me in this regards & answer the following questions:

1. Can i install the the Intel Proset Software & use VLAN & Team feature for this Adapter Intel Pro 1000 MT ?
2. Witch version of  Windows 2008 Series i can use for these features ?
3. Is is any solution possible by upgrading the Adapter firmware

Best regards, Jalal N.

Hi,

         I am aware 82574l is notorious for the link loss when driven with lots of data. I have tried a lot of workarounds suggested in different forums. But couldn’t fix the problem.  I am using tegra based board with 82574l NIC. The OS is Linux Ubuntu 13.04. The e1000e driver was upgraded to 3.04 to rule out any software problems. I am using iperf to reproduce the problem.

The 2 platforms which runs iperf server and client are identical boards with 82574l. I assign them static IP connected to each other using CAT5e cable. From information gathered from forums, I tried disabling ASPM in NIC and kernel. I have also tried disabling TSO, auto negotiation, changed TX/Rx ring buffer sizes, Rx interrups/sec. Forced the three interrupts separately. Tried modifying the 15th bit of the eeprom setting (58->5a which is said to disable Power save). But none of them helped. The problem is there is hardly any error messages which point to the error. Just the netdev watchdog times out Tx and resets the NIC. I tried disabling the watchdog. But the system is stuck.

Attached are the logs (TSO not disabled). Are there any tools, I can use to debug the problem? Any suggestions? The problem occurs randomly sometimes within minutes (but sometime takes hours).

Thanks,

Tony

I connected the Ethernet Server Adaptor X520-DA2 to my linux (Ubuntu 12.04) machine.

The ixgbe driver was automatically loaded and I was able to configure an IP.

I am unable to bring up the card when I have an optical fibre connected to it.

When I connect an Ethernet into one of the card's bays and connect an Ethernet cable, I am able to use the card.

I did run

$ ethtool -p eth2

on the interface but the link is still shown not to be detected.

Do I need any other kernel configurations to make the Optical fibre cable get detected.

Thanks

Attached below are the Kernel configuration file and the dmesg output

Hello

I have been doing different benchmarks using iperf-2.0.5.2-Win32  and NTttcp.exe verion 5.28.   I noticed that only one CPU core is handling the network activity as indicated the Perfmon \Per Processor Network Activity cycles\0, Intel(R) I350 Gigabit Network Connection to 23.  In my case its always core 6 that is in use.  the 22 other cpu always show 0 activity.  This is also visiable by watching the TaskManager Performance tab CPU per core activity during the test. only Core 6 has close to 100% activity.

The network card is an  intel I350-T2 OEM card using ProSet drivers version 19.1.51.0 on a Windows 2012 R2 server.  The hardware is Dell T7500 Workstation with 2 Xeon X5660 CPU and 96 GB of ram.

I have RSS enabled in the Intel. driver settings.

Powershell shows the RSS settings for my primary nNIC.

Name                                            : Ethernet 4

InterfaceDescription                        : Intel(R) I350 Gigabit Network Connection

Enabled                                         : True

NumberOfReceiveQueues                : 8

Profile                                             : NUMA

BaseProcessor: [Group:Number]        0

MaxProcessor: [Group:Number]         :63

MaxProcessors                                 : 8

RssProcessorArray: [Group:Number/NUMA Distance] :

IndirectionTable: [Group:Number]           :

the Intel power shell setting return the following:

PS C:\Users\> Get-IntelNetAdapterSetting

Name                                       DisplayName            DisplayValue    RegistryKeyword         RegistryValue

----                                       -----------            ------------    ---------------         -------------

Intel(R) I350 Gigabit Network Connection   Locally Administere...                 NetworkAddress

Intel(R) I350 Gigabit Network Connection   Low Latency Interru... {}              LLIPorts                {}

Intel(R) I350 Gigabit Network Connection   SR-IOV Queues          6               *NumVFs                 6

Intel(R) I350 Gigabit Network Connection   VM Queues              1               VMQueues                1

Intel(R) I350 Gigabit Network Connection   Receive Buffers        256             *ReceiveBuffers         256

Intel(R) I350 Gigabit Network Connection   RSS Base Processor ... 0               *RssBaseProcNumber      0

Intel(R) I350 Gigabit Network Connection   Transmit Buffers       512             *TransmitBuffers        512

Intel(R) I350 Gigabit Network Connection   Low Latency Interrupts Disabled        EnableLLI               0

Intel(R) I350 Gigabit Network Connection   Flow Control           Rx & Tx Enabled *FlowControl            3

Intel(R) I350 Gigabit Network Connection   Interrupt Moderation   Enabled         *InterruptModeration    1

Intel(R) I350 Gigabit Network Connection   IPv4 Checksum Offload  Rx & Tx Enabled *IPChecksumOffloadIPv4  3

Intel(R) I350 Gigabit Network Connection   Jumbo Packet           Disabled        *JumboPacket            1514

Intel(R) I350 Gigabit Network Connection   Large Send Offload ... Enabled         *LsoV2IPv4              1

Intel(R) I350 Gigabit Network Connection   Large Send Offload ... Enabled         *LsoV2IPv6              1

Intel(R) I350 Gigabit Network Connection   Maximum number of R... 8               *MaxRssProcessors       8

Intel(R) I350 Gigabit Network Connection   Preferred NUMA node    System Default  *NumaNodeId             65535

Intel(R) I350 Gigabit Network Connection   Maximum Number of R... 8 Queues        *NumRssQueues           8

Intel(R) I350 Gigabit Network Connection   ARP Offload            Enabled         *PMARPOffload           1

Intel(R) I350 Gigabit Network Connection   NS Offload             Enabled         *PMNSOffload            1

Intel(R) I350 Gigabit Network Connection   Packet Priority & VLAN Packet Prior... *PriorityVLANTag        3

Intel(R) I350 Gigabit Network Connection   Receive Side Scaling   Enabled         *RSS                    1

Intel(R) I350 Gigabit Network Connection   RSS load balancing ... NUMAScaling     *RSSProfile             3

Intel(R) I350 Gigabit Network Connection   Speed & Duplex         1.0 Gbps Ful... *SpeedDuplex            6

Intel(R) I350 Gigabit Network Connection   SR-IOV                 Enabled         *SRIOV                  1

Intel(R) I350 Gigabit Network Connection   TCP Checksum Offloa... Rx & Tx Enabled *TCPChecksumOffloadIPv4 3

Intel(R) I350 Gigabit Network Connection   TCP Checksum Offloa... Rx & Tx Enabled *TCPChecksumOffloadIPv6 3

Intel(R) I350 Gigabit Network Connection   UDP Checksum Offloa... Rx & Tx Enabled *UDPChecksumOffloadIPv4 3

Intel(R) I350 Gigabit Network Connection   UDP Checksum Offloa... Rx & Tx Enabled *UDPChecksumOffloadIPv6 3

Intel(R) I350 Gigabit Network Connection   Virtual Machine Queues Enabled         *VMQ                    1

Intel(R) I350 Gigabit Network Connection   Wake on Magic Packet   Enabled         *WakeOnMagicPacket      1

Intel(R) I350 Gigabit Network Connection   Wake on Pattern Match  Enabled         *WakeOnPattern          1

Intel(R) I350 Gigabit Network Connection   DMA Coalescing         Disabled        DMACoalescing           0

Intel(R) I350 Gigabit Network Connection   Energy Efficient Et... On if system... EEELinkAdvertisement    1

Intel(R) I350 Gigabit Network Connection   Wake on Magic Packe... Disabled        EnablePME               0

Intel(R) I350 Gigabit Network Connection   Interrupt Moderatio... Adaptive        ITR                     65535

Intel(R) I350 Gigabit Network Connection   Log Link State Event   Enabled         LogLinkStateEvent       51

Intel(R) I350 Gigabit Network Connection   Gigabit Master Slav... Auto Detect     MasterSlave             0

Intel(R) I350 Gigabit Network Connection   Reduce link speed d... Enabled         ReduceSpeedOnPowerDown  1

Intel(R) I350 Gigabit Network Connection   Wait for Link          Auto Detect     WaitAutoNegComplete     2

Intel(R) I350 Gigabit Network Connection   Wake on Link Settings  Disabled        WakeOnLink              0

Another interesting issue is that SMB3 does not detect the I350-T2 NIC as having RSS enabled.

The active Intel I350 intel nic is Interface index 27.

PS C:\Users\> Get-SmbClientNetworkInterface  and output with Get-SmbServerNetworkInterface

Interface Index     RSS Capable         RDMA Capable        Speed               IpAddresses         Friendly Name

---------------     -----------         ------------        -----               -----------         -------------

20                  False               False               0  bps              {fe80::5efe:192.... isatap.{DE460070...

16                  False               False               0  bps              {fe80::5efe:192.... isatap.{F21F7E71...

15                  False               False               0  bps              {fe80::5efe:192.... isatap.{F3FB0818...

14                  False               False               0  bps              {fe80::5efe:192.... isatap.{5993ADFC...

29                  False               False               1 Gbps              {fe80::6987:ecfb... Ethernet 5

27                  False               False               1 Gbps              {fe80::4953:f511... Ethernet 4

23                  False               False               0  bps              {fe80::5efe:10.6... isatap.{651D92EC...

19                  False               False               100 Mbps            {fe80::f8d1:df34... VirtualBox Host-...

25                  False               False               100 Mbps            {fe80::c5b4:ca4e... VMware Network A...

26                  False               False               100 Mbps            {fe80::f976:cf0e... VMware Network A...

9                   False               False               100 Mbps            {fe80::308a:9214... Ethernet 2

Any suggestion on how to fix this issue ?

Naim.

I have a system board with two integrated NICs.  I was able to enable "MonitorMode" to pass 802.1q tagged traffic through to a VMware test installation with a Windows 7 host.  I have since upgraded to Windows 8.1 and the driver for the 82574L series is now listed as being provided by Mircosoft with a prefix of "e1i" - this is not listed on the list of drivers for which to enable either "MonitorMode" or "EnableMonitorMode" on the Intel site.

According the the driver details, this driver was supplied by Intel and is listed as version 12.6.47.1

If I could override the Microsoft driver, I would be in business.  I am sure I can enable MonitorMode on the alternate 82579V integrated NIC, but I want to keep using that as my primary adapter and use the 574L NIC for VM testing.

Is there any way to accomplish this (allowing 802.1q tagged traffic to pass through the 82574L adapter on Windows 8.1 with the default Microsoft drivers, without the tags being stripped)?

(I bought an i350-T2 to resolve this issue but it ended up killing my system's ability to boot unless I removed half the memory, so "install a different NIC" has already been tried )

Thanks!

Windows 7/64, Prowinx64 v19.1 installed. Device manager shows do Advanced Features. The display reports "Your device/operating system combination does not support any VLAN advanced settings."

I'm interested in enabling jumbo packets.

Hi,

I am trying to get tagged packets to work in SRIOV mode.  I am using latest dpdk  1.6 version with an application VM.

The setup is as below: The same configuration works for untagged packets.

Guest VM (Virtual Function)--- > KVM host (PF) -->Client server

When the packet is tagged (vlan tag/id) the packet is not see on the KVM host at all. I tried setting the same vlan tag on the KVM host also - ip link set <port> vf 0 vlan <vlan id> and that didnt work either.

I also tried changing the ol_flags within my application to |= PKT_TX_VLAN_PKT on the egress so the dpdk pmd driver sets the flag IXGBE_ADVTXD_DCMD_VLE in ixgbe_xmit_packets and still don't see the tagged packets going out. It looks like it just doesn't send it out when the tag is present. I am not sure why the host doesnt see the tagged packet at all. Can someone let me know what I am missing?

Should the packet arriving at ixgbe_xmit_packets - have the tag present or not? Do we all the offload flag PKT_TX_VLAN_PKT or not? I have tried different combinations and have been unsuccessful in getting the packet to reach the KVM host. This is true even when the dpdk l2fwd application is used instead of our application.

Any help would be highly appreciated!

Thx

Roopa

I just bought Intel 82574L gigabit network card and thought it has PTP hardware clock due to it supports IEEE1588. However, when I do "ethtool -T p1p1", it says PTP hardware clock is NULL. When I use "phc2sys -s p1p1", it says "Interface p1p1 does not have a PHC". Is it a driver problem, kernel problem or I should get another network card? If I need to get another Network card, which one should I get in order to have PHC?

My kernel is RHEL 3.10

My NIC driver is e1000e 3.0.4-NAPI

Olá eu tenho uma placa intel desktop board dh61BE que eu atualizei para a versao 099 e agora minha máquina nao da nenhum boot ja tentei até por usb ela simplismente trava quando vai dar algum boot, eu consigo acessar tudo na bios mais na hora de dar algum boot para restaurar a versao da bios ou para atualizar para uma mais nova ela simplismente trava...

Poderiam me dar uma solução ???

Obrigado pela atenção!!!

Hi,

This is my first visit/post here to this community so I hope I'm asking the right questions in the right place.

I am using the 82599ES adapter with SR-IOV enabled under Ubuntu 14.04 under kernel 3.13.0-30 but this same behavior was witness under 3.13.0-24.  The corresponding VFs are created and visible in 'lspci' and 'ip link show'.  I am intending to pass VFs to guest OSs in KVM running libvirt 1.2.2 as 'hostdev' devices in the guest's XML configuration file.

Once the VFs are created, I proceed to set VF parameters such as a MAC and VLAN ID.  I believed MACs were to be automatically generated by the driver but perhaps I'm wrong as it doesn't seem to be the case.  So in this particular case, I set the VF MAC to the auto generated MAC that libvirt assigns to the interface by editing the guest through virsh edit.

root@ubuntu:~# ip link set eth4 up

root@ubuntu:~# ip link set eth4 vf 0 mac 52:54:00:4d:8b:bc

root@ubuntu:~# ip link set eth4 vf 0 vlan 100

root@ubuntu:~# ip link set eth4 vf 1 mac 52:54:00:5e:3b:e2

root@ubuntu:~# ip link set eth4 vf 1 vlan 200

root@ubuntu:~# ip link set eth4 vf 2 mac 52:54:00:19:e3:f8

root@ubuntu:~# ip link set eth4 vf 2 vlan 300

root@ubuntu:~# ip link set eth4 vf 3 mac 52:54:00:1c:7e:79

root@ubuntu:~# ip link set eth4 vf 3 vlan 100

root@ubuntu:~# ip link set eth4 vf 4 mac 52:54:00:45:c9:7d

root@ubuntu:~# ip link set eth4 vf 4 vlan 200

root@ubuntu:~# ip link set eth4 vf 5 mac 52:54:00:32:13:0e

root@ubuntu:~# ip link set eth4 vf 5 vlan 300

This results in something like the following (dmesg output below also).

root@ubuntu:~# ip link show dev eth4

11: eth4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000

    link/ether 90:e2:ba:47:2c:30 brd ff:ff:ff:ff:ff:ff

    vf 0 MAC 52:54:00:4d:8b:bc, vlan 100, spoof checking on, link-state auto

    vf 1 MAC 52:54:00:5e:3b:e2, vlan 200, spoof checking on, link-state auto

    vf 2 MAC 52:54:00:19:e3:f8, vlan 300, spoof checking on, link-state auto

    vf 3 MAC 52:54:00:1c:7e:79, vlan 100, spoof checking on, link-state auto

    vf 4 MAC 52:54:00:45:c9:7d, vlan 200, spoof checking on, link-state auto

    vf 5 MAC 52:54:00:32:13:0e, vlan 300, spoof checking on, link-state auto

    vf 6 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

    vf 7 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

    vf 8 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

    vf 9 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

    vf 10 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

    vf 11 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

    vf 12 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

    vf 13 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

    vf 14 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

    vf 15 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

Jul 16 10:29:21 ubuntu kernel: [ 8518.503869] ixgbe 0000:81:00.0: registered PHC device on eth4

Jul 16 10:29:21 ubuntu kernel: [ 8518.608585] IPv6: ADDRCONF(NETDEV_UP): eth4: link is not ready

Jul 16 10:29:21 ubuntu kernel: [ 8518.672821] ixgbe 0000:81:00.0 eth4: detected SFP+: 3

Jul 16 10:29:21 ubuntu kernel: [ 8518.912379] ixgbe 0000:81:00.0 eth4: NIC Link is Up 10 Gbps, Flow Control: RX/TX

Jul 16 10:29:21 ubuntu kernel: [ 8518.912536] IPv6: ADDRCONF(NETDEV_CHANGE): eth4: link becomes ready

Jul 16 10:29:28 ubuntu kernel: [ 8526.350238] ixgbe 0000:81:00.0: setting MAC 52:54:00:4d:8b:bc on VF 0

Jul 16 10:29:28 ubuntu kernel: [ 8526.350244] ixgbe 0000:81:00.0: Reload the VF driver to make this change effective.

Jul 16 10:29:28 ubuntu kernel: [ 8526.363123] ixgbe 0000:81:00.0: Setting VLAN 100, QOS 0x0 on VF 0

Jul 16 10:29:28 ubuntu kernel: [ 8526.387115] ixgbe 0000:81:00.0: setting MAC 52:54:00:5e:3b:e2 on VF 1

Jul 16 10:29:28 ubuntu kernel: [ 8526.387120] ixgbe 0000:81:00.0: Reload the VF driver to make this change effective.

Jul 16 10:29:28 ubuntu kernel: [ 8526.399143] ixgbe 0000:81:00.0: Setting VLAN 200, QOS 0x0 on VF 1

Jul 16 10:29:28 ubuntu kernel: [ 8526.432961] ixgbe 0000:81:00.0: setting MAC 52:54:00:19:e3:f8 on VF 2

Jul 16 10:29:28 ubuntu kernel: [ 8526.432967] ixgbe 0000:81:00.0: Reload the VF driver to make this change effective.

Jul 16 10:29:28 ubuntu kernel: [ 8526.435878] ixgbe 0000:81:00.0: Setting VLAN 300, QOS 0x0 on VF 2

Jul 16 10:29:29 ubuntu kernel: [ 8526.468947] ixgbe 0000:81:00.0: setting MAC 52:54:00:1c:7e:79 on VF 3

Jul 16 10:29:29 ubuntu kernel: [ 8526.468953] ixgbe 0000:81:00.0: Reload the VF driver to make this change effective.

Jul 16 10:29:29 ubuntu kernel: [ 8526.471825] ixgbe 0000:81:00.0: Setting VLAN 100, QOS 0x0 on VF 3

Jul 16 10:29:29 ubuntu kernel: [ 8526.495211] ixgbe 0000:81:00.0: setting MAC 52:54:00:45:c9:7d on VF 4

Jul 16 10:29:29 ubuntu kernel: [ 8526.495217] ixgbe 0000:81:00.0: Reload the VF driver to make this change effective.

Jul 16 10:29:29 ubuntu kernel: [ 8526.561157] ixgbe 0000:81:00.0: Setting VLAN 200, QOS 0x0 on VF 4

Jul 16 10:29:29 ubuntu kernel: [ 8526.564616] ixgbe 0000:81:00.0: setting MAC 52:54:00:32:13:0e on VF 5

Jul 16 10:29:29 ubuntu kernel: [ 8526.564621] ixgbe 0000:81:00.0: Reload the VF driver to make this change effective.

Jul 16 10:29:31 ubuntu kernel: [ 8528.628424] ixgbe 0000:81:00.0: Setting VLAN 300, QOS 0x0 on VF 5

I am passing VF 0-2 to one guest OS and VF 3-5 to a second OS. 

Guest1

Guest2

And then I start the guests.  They boot normal and detect VFs I've passed through but this is where things get hokey.  The guest may sporadically have an all zeros MAC and sometimes not.  But most notably is this difference in the output of 'ip link show dev eth4' compared to before the guest was started.

Before Guest Start

root@ubuntu:~# ip link show dev eth4

11: eth4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000

    link/ether 90:e2:ba:47:2c:30 brd ff:ff:ff:ff:ff:ff

    vf 0 MAC 52:54:00:4d:8b:bc, vlan 100, spoof checking on, link-state auto

    vf 1 MAC 52:54:00:5e:3b:e2, vlan 200, spoof checking on, link-state auto

    vf 2 MAC 52:54:00:19:e3:f8, vlan 300, spoof checking on, link-state auto

    vf 3 MAC 52:54:00:1c:7e:79, vlan 100, spoof checking on, link-state auto

    vf 4 MAC 52:54:00:45:c9:7d, vlan 200, spoof checking on, link-state auto

    vf 5 MAC 52:54:00:32:13:0e, vlan 300, spoof checking on, link-state auto

    vf 6 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

    vf 7 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

    vf 8 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

    vf 9 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

    vf 10 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

    vf 11 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

    vf 12 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

    vf 13 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

    vf 14 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

    vf 15 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

After Guest Start

root@ubuntu:~# ip link show dev eth4

11: eth4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000

    link/ether 90:e2:ba:47:2c:30 brd ff:ff:ff:ff:ff:ff

    vf 0 MAC 52:54:00:4d:8b:bc, spoof checking on, link-state auto

    vf 1 MAC 52:54:00:19:e3:f8, spoof checking on, link-state auto

    vf 2 MAC 52:54:00:45:c9:7d, spoof checking on, link-state auto

    vf 3 MAC 52:54:00:1c:7e:79, vlan 100, spoof checking on, link-state auto

    vf 4 MAC 52:54:00:45:c9:7d, vlan 200, spoof checking on, link-state auto

    vf 5 MAC 52:54:00:32:13:0e, vlan 300, spoof checking on, link-state auto

    vf 6 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

    vf 7 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

    vf 8 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

    vf 9 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

    vf 10 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

    vf 11 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

    vf 12 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

    vf 13 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

    vf 14 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

    vf 15 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

The VLAN ID is missing for VF0-2.  VF1 has VF2's original MAC and VF2's MAC has become VF4's MAC.

Here is the resulting dmesg output from starting my guests.

Jul 16 10:37:56 ubuntu kernel: [ 9034.298337] pci-stub 0000:81:10.0: claimed by stub

Jul 16 10:37:56 ubuntu kernel: [ 9034.298359] pci-stub 0000:81:10.2: claimed by stub

Jul 16 10:37:56 ubuntu kernel: [ 9034.298369] pci-stub 0000:81:10.4: claimed by stub

Jul 16 10:37:56 ubuntu kernel: [ 9034.298378] pci-stub 0000:81:10.6: claimed by stub

Jul 16 10:37:56 ubuntu kernel: [ 9034.298385] pci-stub 0000:81:11.0: claimed by stub

Jul 16 10:37:56 ubuntu kernel: [ 9034.298411] pci-stub 0000:81:11.2: claimed by stub

Jul 16 10:37:56 ubuntu kernel: [ 9034.298428] pci-stub 0000:81:11.4: claimed by stub

Jul 16 10:37:56 ubuntu kernel: [ 9034.298439] pci-stub 0000:81:11.6: claimed by stub

Jul 16 10:37:56 ubuntu kernel: [ 9034.298453] pci-stub 0000:81:12.0: claimed by stub

Jul 16 10:37:56 ubuntu kernel: [ 9034.298466] pci-stub 0000:81:12.2: claimed by stub

Jul 16 10:37:56 ubuntu kernel: [ 9034.298478] pci-stub 0000:81:12.4: claimed by stub

Jul 16 10:37:56 ubuntu kernel: [ 9034.298489] pci-stub 0000:81:12.6: claimed by stub

Jul 16 10:37:56 ubuntu kernel: [ 9034.298499] pci-stub 0000:81:13.0: claimed by stub

Jul 16 10:37:56 ubuntu kernel: [ 9034.298509] pci-stub 0000:81:13.2: claimed by stub

Jul 16 10:37:56 ubuntu kernel: [ 9034.298519] pci-stub 0000:81:13.4: claimed by stub

Jul 16 10:37:56 ubuntu kernel: [ 9034.298528] pci-stub 0000:81:13.6: claimed by stub

Jul 16 10:37:56 ubuntu kernel: [ 9034.298539] pci-stub 0000:81:10.1: claimed by stub

Jul 16 10:37:56 ubuntu kernel: [ 9034.298550] pci-stub 0000:81:10.3: claimed by stub

Jul 16 10:37:56 ubuntu kernel: [ 9034.298561] pci-stub 0000:81:10.5: claimed by stub

Jul 16 10:37:56 ubuntu kernel: [ 9034.298571] pci-stub 0000:81:10.7: claimed by stub

Jul 16 10:37:56 ubuntu kernel: [ 9034.298580] pci-stub 0000:81:11.1: claimed by stub

Jul 16 10:37:56 ubuntu kernel: [ 9034.298591] pci-stub 0000:81:11.3: claimed by stub

Jul 16 10:37:56 ubuntu kernel: [ 9034.298601] pci-stub 0000:81:11.5: claimed by stub

Jul 16 10:37:56 ubuntu kernel: [ 9034.298611] pci-stub 0000:81:11.7: claimed by stub

Jul 16 10:37:56 ubuntu kernel: [ 9034.298621] pci-stub 0000:81:12.1: claimed by stub

Jul 16 10:37:56 ubuntu kernel: [ 9034.298633] pci-stub 0000:81:12.3: claimed by stub

Jul 16 10:37:56 ubuntu kernel: [ 9034.298643] pci-stub 0000:81:12.5: claimed by stub

Jul 16 10:37:56 ubuntu kernel: [ 9034.298653] pci-stub 0000:81:12.7: claimed by stub

Jul 16 10:37:56 ubuntu kernel: [ 9034.298662] pci-stub 0000:81:13.1: claimed by stub

Jul 16 10:37:56 ubuntu kernel: [ 9034.298672] pci-stub 0000:81:13.3: claimed by stub

Jul 16 10:37:56 ubuntu kernel: [ 9034.298683] pci-stub 0000:81:13.5: claimed by stub

Jul 16 10:37:56 ubuntu kernel: [ 9034.298693] pci-stub 0000:81:13.7: claimed by stub

Jul 16 10:37:56 ubuntu kernel: [ 9034.324752] ixgbe 0000:81:00.0: setting MAC 52:54:00:4d:8b:bc on VF 0

Jul 16 10:37:56 ubuntu kernel: [ 9034.324759] ixgbe 0000:81:00.0: Reload the VF driver to make this change effective.

Jul 16 10:37:56 ubuntu kernel: [ 9034.344723] ixgbe 0000:81:00.1: setting MAC 52:54:00:5e:3b:e2 on VF 0

Jul 16 10:37:56 ubuntu kernel: [ 9034.344730] ixgbe 0000:81:00.1: Reload the VF driver to make this change effective.

Jul 16 10:37:56 ubuntu kernel: [ 9034.344733] ixgbe 0000:81:00.1: The VF MAC address has been set, but the PF device is not up.

Jul 16 10:37:56 ubuntu kernel: [ 9034.344736] ixgbe 0000:81:00.1: Bring the PF device up before attempting to use the VF device.

Jul 16 10:37:56 ubuntu kernel: [ 9034.360702] ixgbe 0000:81:00.0: setting MAC 52:54:00:19:e3:f8 on VF 1

Jul 16 10:37:56 ubuntu kernel: [ 9034.360708] ixgbe 0000:81:00.0: Reload the VF driver to make this change effective.

Jul 16 10:37:56 ubuntu kernel: [ 9034.665913] audit_printk_skb: 69 callbacks suppressed

Jul 16 10:37:56 ubuntu kernel: [ 9034.665915] type=1400 audit(1405532276.635:35): apparmor="STATUS" operation="profile_load" profile="unconfined" name="libvirt-b1ce4fac-83d7-3f00-8127-07ab6f1e15e3" pid=2779 comm="apparmor_parser"

Jul 16 10:37:56 ubuntu kernel: [ 9034.880720] pci-stub 0000:81:10.0: enabling device (0000 -> 0002)

Jul 16 10:37:58 ubuntu kernel: [ 9036.454955] pci-stub 0000:81:10.0: kvm assign device

Jul 16 10:37:58 ubuntu kernel: [ 9036.456580] pci-stub 0000:81:10.1: enabling device (0000 -> 0002)

Jul 16 10:37:58 ubuntu kernel: [ 9036.558625] pci-stub 0000:81:10.1: kvm assign device

Jul 16 10:37:58 ubuntu kernel: [ 9036.559693] pci-stub 0000:81:10.2: enabling device (0000 -> 0002)

Jul 16 10:37:58 ubuntu kernel: [ 9036.662765] pci-stub 0000:81:10.2: kvm assign device

Jul 16 10:38:22 ubuntu kernel: [ 9060.288671] kvm: zapping shadow pages for mmio generation wraparound

Jul 16 10:38:22 ubuntu kernel: [ 9060.882172] ixgbe 0000:81:00.0 eth4: VF Reset msg received from vf 0

Jul 16 10:38:22 ubuntu kernel: [ 9060.893544] pci-stub 0000:81:10.0: irq 193 for MSI/MSI-X

Jul 16 10:38:22 ubuntu kernel: [ 9060.921582] pci-stub 0000:81:10.0: irq 193 for MSI/MSI-X

Jul 16 10:38:22 ubuntu kernel: [ 9060.921601] pci-stub 0000:81:10.0: irq 194 for MSI/MSI-X

Jul 16 10:38:22 ubuntu kernel: [ 9060.945515] pci-stub 0000:81:10.0: irq 193 for MSI/MSI-X

Jul 16 10:38:22 ubuntu kernel: [ 9060.945528] pci-stub 0000:81:10.0: irq 194 for MSI/MSI-X

Jul 16 10:38:23 ubuntu kernel: [ 9061.053636] pci-stub 0000:81:10.1: irq 195 for MSI/MSI-X

Jul 16 10:38:23 ubuntu kernel: [ 9061.073718] pci-stub 0000:81:10.1: irq 195 for MSI/MSI-X

Jul 16 10:38:23 ubuntu kernel: [ 9061.073732] pci-stub 0000:81:10.1: irq 196 for MSI/MSI-X

Jul 16 10:38:23 ubuntu kernel: [ 9061.101789] pci-stub 0000:81:10.1: irq 195 for MSI/MSI-X

Jul 16 10:38:23 ubuntu kernel: [ 9061.101803] pci-stub 0000:81:10.1: irq 196 for MSI/MSI-X

Jul 16 10:38:23 ubuntu kernel: [ 9061.187991] ixgbe 0000:81:00.0 eth4: VF Reset msg received from vf 1

Jul 16 10:38:23 ubuntu kernel: [ 9061.201800] pci-stub 0000:81:10.2: irq 197 for MSI/MSI-X

Jul 16 10:38:23 ubuntu kernel: [ 9061.225840] pci-stub 0000:81:10.2: irq 197 for MSI/MSI-X

Jul 16 10:38:23 ubuntu kernel: [ 9061.225863] pci-stub 0000:81:10.2: irq 198 for MSI/MSI-X

Jul 16 10:38:23 ubuntu kernel: [ 9061.261958] pci-stub 0000:81:10.2: irq 197 for MSI/MSI-X

Jul 16 10:38:23 ubuntu kernel: [ 9061.261972] pci-stub 0000:81:10.2: irq 198 for MSI/MSI-X

Jul 16 10:38:31 ubuntu kernel: [ 9069.898230] ixgbe 0000:81:00.0 eth4: VF Reset msg received from vf 0

Jul 16 10:38:42 ubuntu kernel: [ 9080.567505] ixgbe 0000:81:00.1: setting MAC 52:54:00:1c:7e:79 on VF 1

Jul 16 10:38:42 ubuntu kernel: [ 9080.567510] ixgbe 0000:81:00.1: Reload the VF driver to make this change effective.

Jul 16 10:38:42 ubuntu kernel: [ 9080.567512] ixgbe 0000:81:00.1: The VF MAC address has been set, but the PF device is not up.

Jul 16 10:38:42 ubuntu kernel: [ 9080.567513] ixgbe 0000:81:00.1: Bring the PF device up before attempting to use the VF device.

Jul 16 10:38:42 ubuntu kernel: [ 9080.583695] ixgbe 0000:81:00.0: setting MAC 52:54:00:45:c9:7d on VF 2

Jul 16 10:38:42 ubuntu kernel: [ 9080.583702] ixgbe 0000:81:00.0: Reload the VF driver to make this change effective.

Jul 16 10:38:42 ubuntu kernel: [ 9080.595660] ixgbe 0000:81:00.1: setting MAC 52:54:00:32:13:0e on VF 2

Jul 16 10:38:42 ubuntu kernel: [ 9080.595666] ixgbe 0000:81:00.1: Reload the VF driver to make this change effective.

Jul 16 10:38:42 ubuntu kernel: [ 9080.595669] ixgbe 0000:81:00.1: The VF MAC address has been set, but the PF device is not up.

Jul 16 10:38:42 ubuntu kernel: [ 9080.595672] ixgbe 0000:81:00.1: Bring the PF device up before attempting to use the VF device.

Jul 16 10:38:42 ubuntu kernel: [ 9080.866600] type=1400 audit(1405532322.783:36): apparmor="STATUS" operation="profile_load" profile="unconfined" name="libvirt-ed80fbe8-a135-6d45-1457-5d9d00f1d5ce" pid=2805 comm="apparmor_parser"

Jul 16 10:38:42 ubuntu kernel: [ 9081.052820] pci-stub 0000:81:10.3: enabling device (0000 -> 0002)

Jul 16 10:38:44 ubuntu kernel: [ 9082.503332] pci-stub 0000:81:10.3: kvm assign device

Jul 16 10:38:44 ubuntu kernel: [ 9082.504917] pci-stub 0000:81:10.4: enabling device (0000 -> 0002)

Jul 16 10:38:44 ubuntu kernel: [ 9082.605369] pci-stub 0000:81:10.4: kvm assign device

Jul 16 10:38:44 ubuntu kernel: [ 9082.606482] pci-stub 0000:81:10.5: enabling device (0000 -> 0002)

Jul 16 10:38:44 ubuntu kernel: [ 9082.709469] pci-stub 0000:81:10.5: kvm assign device

Jul 16 10:39:08 ubuntu kernel: [ 9106.462615] kvm: zapping shadow pages for mmio generation wraparound

Jul 16 10:39:08 ubuntu kernel: [ 9107.064370] pci-stub 0000:81:10.3: irq 199 for MSI/MSI-X

Jul 16 10:39:08 ubuntu kernel: [ 9107.088349] pci-stub 0000:81:10.3: irq 199 for MSI/MSI-X

Jul 16 10:39:08 ubuntu kernel: [ 9107.088364] pci-stub 0000:81:10.3: irq 200 for MSI/MSI-X

Jul 16 10:39:09 ubuntu kernel: [ 9107.108372] pci-stub 0000:81:10.3: irq 199 for MSI/MSI-X

Jul 16 10:39:09 ubuntu kernel: [ 9107.108387] pci-stub 0000:81:10.3: irq 200 for MSI/MSI-X

Jul 16 10:39:09 ubuntu kernel: [ 9107.209645] ixgbe 0000:81:00.0 eth4: VF Reset msg received from vf 2

Jul 16 10:39:09 ubuntu kernel: [ 9107.224472] pci-stub 0000:81:10.4: irq 201 for MSI/MSI-X

Jul 16 10:39:09 ubuntu kernel: [ 9107.252548] pci-stub 0000:81:10.4: irq 201 for MSI/MSI-X

Jul 16 10:39:09 ubuntu kernel: [ 9107.252563] pci-stub 0000:81:10.4: irq 202 for MSI/MSI-X

Jul 16 10:39:09 ubuntu kernel: [ 9107.272562] pci-stub 0000:81:10.4: irq 201 for MSI/MSI-X

Jul 16 10:39:09 ubuntu kernel: [ 9107.272578] pci-stub 0000:81:10.4: irq 202 for MSI/MSI-X

Jul 16 10:39:09 ubuntu kernel: [ 9107.364623] pci-stub 0000:81:10.5: irq 203 for MSI/MSI-X

Jul 16 10:39:09 ubuntu kernel: [ 9107.384680] pci-stub 0000:81:10.5: irq 203 for MSI/MSI-X

Jul 16 10:39:09 ubuntu kernel: [ 9107.384695] pci-stub 0000:81:10.5: irq 204 for MSI/MSI-X

Jul 16 10:39:09 ubuntu kernel: [ 9107.404698] pci-stub 0000:81:10.5: irq 203 for MSI/MSI-X

Jul 16 10:39:09 ubuntu kernel: [ 9107.404713] pci-stub 0000:81:10.5: irq 204 for MSI/MSI-X

Jul 16 10:39:28 ubuntu kernel: [ 9126.991794] ixgbe 0000:81:00.0 eth4: VF Reset msg received from vf 2

root@ubuntu:~# cat /etc/modprobe.d/ixgbe.conf

options ixgbe max_vfs=16

options loop max_loop=128

blacklist ixgbevf

I removed the module and re-added to provide the dmesg output here.

root@ubuntu:~# modprobe ixgbe

Jul 16 10:20:31 ubuntu kernel: [ 7988.720851] ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - version 3.15.1-k

Jul 16 10:20:31 ubuntu kernel: [ 7988.720856] ixgbe: Copyright (c) 1999-2013 Intel Corporation.

Jul 16 10:20:31 ubuntu kernel: [ 7988.721376] ixgbe 0000:81:00.0: Enabling SR-IOV VFs using the max_vfs module parameter is deprecated - please use the pci sysfs interface instead.

Jul 16 10:20:32 ubuntu kernel: [ 7988.952543] pci 0000:81:10.0: [8086:10ed] type 00 class 0x020000

Jul 16 10:20:32 ubuntu kernel: [ 7988.952824] pci 0000:81:10.2: [8086:10ed] type 00 class 0x020000

Jul 16 10:20:32 ubuntu kernel: [ 7988.953029] pci 0000:81:10.4: [8086:10ed] type 00 class 0x020000

Jul 16 10:20:32 ubuntu kernel: [ 7988.953200] pci 0000:81:10.6: [8086:10ed] type 00 class 0x020000

Jul 16 10:20:32 ubuntu kernel: [ 7988.953367] pci 0000:81:11.0: [8086:10ed] type 00 class 0x020000

Jul 16 10:20:32 ubuntu kernel: [ 7988.953533] pci 0000:81:11.2: [8086:10ed] type 00 class 0x020000

Jul 16 10:20:32 ubuntu kernel: [ 7988.953698] pci 0000:81:11.4: [8086:10ed] type 00 class 0x020000

Jul 16 10:20:32 ubuntu kernel: [ 7988.953870] pci 0000:81:11.6: [8086:10ed] type 00 class 0x020000

Jul 16 10:20:32 ubuntu kernel: [ 7988.954039] pci 0000:81:12.0: [8086:10ed] type 00 class 0x020000

Jul 16 10:20:32 ubuntu kernel: [ 7988.954214] pci 0000:81:12.2: [8086:10ed] type 00 class 0x020000

Jul 16 10:20:32 ubuntu kernel: [ 7988.954400] pci 0000:81:12.4: [8086:10ed] type 00 class 0x020000

Jul 16 10:20:32 ubuntu kernel: [ 7988.954629] pci 0000:81:12.6: [8086:10ed] type 00 class 0x020000

Jul 16 10:20:32 ubuntu kernel: [ 7988.954805] pci 0000:81:13.0: [8086:10ed] type 00 class 0x020000

Jul 16 10:20:32 ubuntu kernel: [ 7988.954976] pci 0000:81:13.2: [8086:10ed] type 00 class 0x020000

Jul 16 10:20:32 ubuntu kernel: [ 7988.955148] pci 0000:81:13.4: [8086:10ed] type 00 class 0x020000

Jul 16 10:20:32 ubuntu kernel: [ 7988.955400] pci 0000:81:13.6: [8086:10ed] type 00 class 0x020000

Jul 16 10:20:32 ubuntu kernel: [ 7988.955842] ixgbe 0000:81:00.0 (unregistered net_device): SR-IOV enabled with 16 VFs

Jul 16 10:20:32 ubuntu kernel: [ 7988.979951] ixgbe 0000:81:00.0: irq 173 for MSI/MSI-X

Jul 16 10:20:32 ubuntu kernel: [ 7988.979989] ixgbe 0000:81:00.0: irq 174 for MSI/MSI-X

Jul 16 10:20:32 ubuntu kernel: [ 7988.980011] ixgbe 0000:81:00.0: Multiqueue Disabled: Rx Queue count = 1, Tx Queue count = 1

Jul 16 10:20:32 ubuntu kernel: [ 7988.980144] ixgbe 0000:81:00.0: PCI Express bandwidth of 32GT/s available

Jul 16 10:20:32 ubuntu kernel: [ 7988.980148] ixgbe 0000:81:00.0: (Speed:5.0GT/s, Width: x8, Encoding Loss:20%)

Jul 16 10:20:32 ubuntu kernel: [ 7988.980234] ixgbe 0000:81:00.0: MAC: 2, PHY: 9, SFP+: 3, PBA No: E66560-006

Jul 16 10:20:32 ubuntu kernel: [ 7988.980237] ixgbe 0000:81:00.0: 90:e2:ba:47:2c:30

Jul 16 10:20:32 ubuntu kernel: [ 7988.982095] ixgbe 0000:81:00.0 eth4: IOV is enabled with 16 VFs

Jul 16 10:20:32 ubuntu kernel: [ 7988.982253] ixgbe 0000:81:00.0: Intel(R) 10 Gigabit Network Connection

Jul 16 10:20:32 ubuntu kernel: [ 7988.982599] ixgbe 0000:81:00.1: Enabling SR-IOV VFs using the max_vfs module parameter is deprecated - please use the pci sysfs interface instead.

Jul 16 10:20:32 ubuntu kernel: [ 7989.212846] pci 0000:81:10.1: [8086:10ed] type 00 class 0x020000

Jul 16 10:20:32 ubuntu kernel: [ 7989.213062] pci 0000:81:10.3: [8086:10ed] type 00 class 0x020000

Jul 16 10:20:32 ubuntu kernel: [ 7989.213161] pci 0000:81:10.5: [8086:10ed] type 00 class 0x020000

Jul 16 10:20:32 ubuntu kernel: [ 7989.213257] pci 0000:81:10.7: [8086:10ed] type 00 class 0x020000

Jul 16 10:20:32 ubuntu kernel: [ 7989.213354] pci 0000:81:11.1: [8086:10ed] type 00 class 0x020000

Jul 16 10:20:32 ubuntu kernel: [ 7989.213451] pci 0000:81:11.3: [8086:10ed] type 00 class 0x020000

Jul 16 10:20:32 ubuntu kernel: [ 7989.213547] pci 0000:81:11.5: [8086:10ed] type 00 class 0x020000

Jul 16 10:20:32 ubuntu kernel: [ 7989.213713] pci 0000:81:11.7: [8086:10ed] type 00 class 0x020000

Jul 16 10:20:32 ubuntu kernel: [ 7989.214277] pci 0000:81:12.1: [8086:10ed] type 00 class 0x020000

Jul 16 10:20:32 ubuntu kernel: [ 7989.214454] pci 0000:81:12.3: [8086:10ed] type 00 class 0x020000

Jul 16 10:20:32 ubuntu kernel: [ 7989.214607] pci 0000:81:12.5: [8086:10ed] type 00 class 0x020000

Jul 16 10:20:32 ubuntu kernel: [ 7989.214740] pci 0000:81:12.7: [8086:10ed] type 00 class 0x020000

Jul 16 10:20:32 ubuntu kernel: [ 7989.214924] pci 0000:81:13.1: [8086:10ed] type 00 class 0x020000

Jul 16 10:20:32 ubuntu kernel: [ 7989.215088] pci 0000:81:13.3: [8086:10ed] type 00 class 0x020000

Jul 16 10:20:32 ubuntu kernel: [ 7989.215222] pci 0000:81:13.5: [8086:10ed] type 00 class 0x020000

Jul 16 10:20:32 ubuntu kernel: [ 7989.215398] pci 0000:81:13.7: [8086:10ed] type 00 class 0x020000

Jul 16 10:20:32 ubuntu kernel: [ 7989.215567] ixgbe 0000:81:00.1 (unregistered net_device): SR-IOV enabled with 16 VFs

Jul 16 10:20:32 ubuntu kernel: [ 7989.239728] ixgbe 0000:81:00.1: irq 175 for MSI/MSI-X

Jul 16 10:20:32 ubuntu kernel: [ 7989.239767] ixgbe 0000:81:00.1: irq 176 for MSI/MSI-X

Jul 16 10:20:32 ubuntu kernel: [ 7989.239793] ixgbe 0000:81:00.1: Multiqueue Disabled: Rx Queue count = 1, Tx Queue count = 1

Jul 16 10:20:32 ubuntu kernel: [ 7989.239930] ixgbe 0000:81:00.1: PCI Express bandwidth of 32GT/s available

Jul 16 10:20:32 ubuntu kernel: [ 7989.239934] ixgbe 0000:81:00.1: (Speed:5.0GT/s, Width: x8, Encoding Loss:20%)

Jul 16 10:20:32 ubuntu kernel: [ 7989.240020] ixgbe 0000:81:00.1: MAC: 2, PHY: 9, SFP+: 4, PBA No: E66560-006

Jul 16 10:20:32 ubuntu kernel: [ 7989.240023] ixgbe 0000:81:00.1: 90:e2:ba:47:2c:31

Jul 16 10:20:32 ubuntu kernel: [ 7989.242601] ixgbe 0000:81:00.1 eth5: IOV is enabled with 16 VFs

Jul 16 10:20:32 ubuntu kernel: [ 7989.242852] ixgbe 0000:81:00.1: Intel(R) 10 Gigabit Network Connection

root@ubuntu:~# lspci | grep 82599

81:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)

81:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)

81:10.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

81:10.1 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

81:10.2 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

81:10.3 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

81:10.4 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

81:10.5 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

81:10.6 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

81:10.7 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

81:11.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

81:11.1 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

81:11.2 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

81:11.3 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

81:11.4 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

81:11.5 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

81:11.6 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

81:11.7 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

81:12.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

81:12.1 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

81:12.2 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

81:12.3 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

81:12.4 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

81:12.5 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

81:12.6 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

81:12.7 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

81:13.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

81:13.1 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

81:13.2 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

81:13.3 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

81:13.4 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

81:13.5 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

81:13.6 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

81:13.7 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

A few additional notes on what I've done.  I've taken the kernel as mentioned before from 3.13.0-24 to 3.13.0-30.

I am using the packaged driver 3.15.1-k in Ubuntu 14.04 as it appears there is issues with compiling the latest driver on 14.04 according to bug #415 and #421, so I have been unsucesfull there.

I've upgrade iproute2 from 3.12.0-2 to 3.15.0

I've performed SR-IOV using another vendor's card under Ubuntu 14.04 and same versions of libvirt and iproute2 and didn't experience this behavior so for that reason I've started here.

Any help or ideas on why things appear to get scrabled is appreciated. 

Thanks!

JC