Multi-port Ethernet in PetaLinux

Many FPGA-based embedded designs require connections to multiple Ethernet devices such as IP cameras, and control of those devices under an operating system, typically Linux. The development of such applications can be accelerated through the use of development boards such as the ZedBoard and the Ethernet FMC. In this tutorial, we will build a custom version of PetaLinux for the ZedBoard and bring up 4 extra Ethernet ports, made available by the Ethernet FMC. The Vivado hardware design used in this tutorial will be very similar to the one we created in a previous tutorial titled: Using AXI Ethernet Subsystem and GMII-to-RGMII in a Multi-port Ethernet design. You don’t need to have followed that tutorial to do this one, as the Vivado project can be built from the sources on Github.

Requirements

To complete this tutorial you will need the following:

• Vivado 2015.4
• PetaLinux SDK 2015.4
• ZedBoard
• Ethernet FMC
• Putty (or similar terminal program)

Tool Setup for Windows users

PetaLinux SDK 2015.4 only runs in the Linux operating system, so Windows users (like me) have to have two machines to follow this tutorial. You can either have two physical machines, which is how I work, or you can have one Windows machine and one Linux virtual machine. In this tutorial, I will assume that you have two physical machines, one running Windows and the other running Linux. My personal setup uses Windows 7 and Ubuntu 14.04 LTS on two separate machines.

If you are building your Linux setup for the first time, here are the supported OSes according to the PetaLinux SDK Installation guide:

• RHEL 5 (32-bit or 64-bit)
• RHEL 6 (32-bit or 64-bit)
• SUSE Enterprise 11 (32-bit or 64-bit)

Note: I had problems installing PetaLinux SDK 2015.4 on 32-bit Ubuntu, as did others, so I use 64-bit Ubuntu and I haven’t had any problems with my setup.

Regenerate the Vivado project

The details of the Vivado design will not be covered by this tutorial as it has already been covered in a previous tutorial - except that in this tutorial, we will be using AXI Ethernet Subsystem IP for all 4 ports. Follow these instructions to regenerate the Vivado project from scripts. Please note that the Git repository is regularly updated for the latest version of Vivado, so you must download the last “commit” for the version of Vivado that you are using.

1. Download the sources from Github here: https://github.com/fpgadeveloper/zedboard-qgige-axieth
2. Depending on your operating system:
   • If you are using a Windows machine, open Windows Explorer, browse to the “Vivado” folder within the sources you just downloaded. Double-click on the build.bat file to run the batch file.
   • If you are using a Linux machine, run Vivado and then select Window->Tcl Console from the welcome screen. In the Tcl console, use the “cd” command to navigate to the “Vivado” folder within the sources you just downloaded. Then type source build.tcl to run the build script.
3. Once the script has finished running, the Vivado project should be regenerated and located in the “Vivado” folder. Run Vivado and open the newly generated project.

Generate the bitstream

The first thing we’ll need to do is to generate the bitstream from the Vivado project.

1. Open the project in Vivado.
2. From the Flow Navigator, click “Generate Bitstream”.
3. Depending on your machine, it will take several minutes to perform synthesis and implementation. In the end, you should see the following message. Just select “View Reports” and click OK.
4. Now we need to use the “Export to SDK” feature to create a hardware description file (.hdf) for the project. From the menu, select File->Export->Export Hardware.
5. In the Export Hardware window, tick “Include bitstream” and choose “Local to Project” as the export location.

Build PetaLinux for our design

Now it’s time to move to our Linux machine and use the PetaLinux SDK to build PetaLinux for our hardware design.

1. On your Linux machine, start a command terminal.

2. Type source /<your-petalinux-install-dir>/settings.sh into the terminal and press Enter. Obviously you must insert the location of your PetaLinux installation.

3. For consistency, let’s work from a directory called projects/zedboard-multiport-ethernet in your home directory. Create that directory and then “cd” to it.

4. Use a USB stick or another method to copy the entire Vivado project directory (should be zedboard_qgige_axieth) from your Windows machine onto your Linux machine. Place it into the directory we just created.

5. Create a PetaLinux project using this command: petalinux-create --type project --template zynq --name petalinux_prj

6. Change to the “petalinux_prj” directory in the command terminal.

   Stay in the PetaLinux project folder from here on. It is important that all the following commands are run from the PetaLinux project folder that we just created.

7. Import the Vivado generated hardware description into our PetaLinux project with the command: petalinux-config --get-hw-description ../zedboard_qgige_axieth/zedboard_qgige_axieth.sdk/

8. The Linux System Configuration will open, but we don’t have any changes to make here, so simply exit and save the configuration.

9. Configure the Linux kernel with the command: petalinux-config -c kernel

10. In the Kernel configuration, we need to disable the Xilinx AXI DMA driver, as it conflicts with the AXI Ethernet driver. Disable: Device Drivers->DMA Engine support->Xilinx DMA Engines->Xilinx AXI DMA Engine, then exit and save the configuration.

    For newer versions of PetaLinux: In the more recent versions of PetaLinux, we no longer have to disable the Xilinx AXI DMA Engine.
    

11. We don’t have anything to change in the Linux root file system, but if you want to make your own changes, run the command: petalinux-config -c rootfs

12. The device tree that was generated by PetaLinux SDK will not contain the MAC addresses, nor the addresses of the Ethernet PHYs, so we have to add this information manually. Open the system-top.dts file in the petalinux-prj/subsystems/linux/configs/device-tree directory.

    For newer versions of PetaLinux: In the more recent versions of PetaLinux, we instead have to add this device tree code to the system-user.dtsi file located under project-spec/meta-user/recipes-bsp/device-tree/files.
    

13. Add the following code to the end of the system-top.dts file and then save it:

&axi_ethernet_0 {
    local-mac-address = [00 0a 35 00 01 22];
    phy-handle = <&phy0>;
    xlnx,has-mdio = <0x1>;
    phy-mode = "rgmii-rxid";
    mdio {
        #address-cells = <1>;
        #size-cells = <0>;
        phy0: phy@0 {
            compatible = "marvell,88e1510";
            device_type = "ethernet-phy";
            reg = <0>;
        };
    };
};

&axi_ethernet_1 {
    local-mac-address = [00 0a 35 00 01 23];
    phy-handle = <&phy1>;
    xlnx,has-mdio = <0x1>;
    phy-mode = "rgmii-rxid";
    mdio {
        #address-cells = <1>;
        #size-cells = <0>;
        phy1: phy@0 {
            compatible = "marvell,88e1510";
            device_type = "ethernet-phy";
            reg = <0>;
        };
    };
};

&axi_ethernet_2 {
    local-mac-address = [00 0a 35 00 01 24];
    phy-handle = <&phy2>;
    xlnx,has-mdio = <0x1>;
    phy-mode = "rgmii-rxid";
    mdio {
        #address-cells = <1>;
        #size-cells = <0>;
        phy2: phy@0 {
            compatible = "marvell,88e1510";
            device_type = "ethernet-phy";
            reg = <0>;
        };
    };
};

&axi_ethernet_3{
    local-mac-address = [00 0a 35 00 01 25];
    phy-handle = <&phy3>;
    xlnx,has-mdio = <0x1>;
    phy-mode = "rgmii-rxid";
    mdio {
        #address-cells = <1>;
        #size-cells = <0>;
        phy3: phy@0 {
            compatible = "marvell,88e1510";
            device_type = "ethernet-phy";
            reg = <0>;
        };
    };
};

1. Build PetaLinux using the command: petalinux-build

PetaLinux will take a few minutes to build depending on your machine.

Boot PetaLinux from SD card

Now we will generate the boot files for an SD card, copy those files to the SD card and then boot PetaLinux on the ZedBoard.

1. Generate the boot files using these commands:
   • petalinux-package --boot --fsbl ./images/linux/zynq_fsbl.elf --fpga ../zedboard_qgige_axieth/zedboard_qgige_axieth.runs/impl_1/design_1_wrapper.bit --uboot --force
   • petalinux-package --prebuilt --fpga ../zedboard_qgige_axieth/zedboard_qgige_axieth.runs/impl_1/design_1_wrapper.bit
2. The boot files can now be found in the petalinux-prj/images/linux folder. Copy the BOOT.BIN and image.ub files into the root of your SD card.
3. Plug the SD card into your ZedBoard.
4. Make sure that your ZedBoard is configured to boot from the SD card by setting jumpers JP7, JP8, JP9, JP10 and JP11 to 00110 respectively.
5. Make sure that a USB cable connects the ZedBoard USB-UART to your PC.
6. Turn ON the ZedBoard.
7. Find the COM port associated with your ZedBoard USB-UART by going into Device Manager.
8. Open a new session in Putty using these settings and the COM port you just identified:
   • Baud rate: 115200bps
   • Data: 8 bits
   • Parity: None
   • Stop bits: 1
9. Watch PetaLinux booting up in the Putty console and wait for the login. If you don’t see anything, you probably missed the boot sequence - just press ENTER and you should see the login prompt.

If you want to see the complete boot log, click here.

Configure the Ethernet ports

Our Vivado design has 5 Ethernet ports: the on-board port of the ZedBoard plus the 4 ports of the Ethernet FMC. In PetaLinux, these ports will be assigned to eth0 (on-board port), and eth1-eth4 (Ethernet FMC ports 0-3). Using ifconfig, we will configure the Ethernet FMC ports with fixed IP addresses. We will then connect one of them to a PC and use ping to test it.

1. First login to PetaLinux using the username “root” and the password “root”.
2. Configure the Ethernet ports using the following commands:
   • ifconfig eth1 192.168.1.11 netmask 255.255.255.0 up
   • ifconfig eth2 192.168.1.12 netmask 255.255.255.0 up
   • ifconfig eth3 192.168.1.13 netmask 255.255.255.0 up
   • ifconfig eth4 192.168.1.14 netmask 255.255.255.0 up
3. When you enter each of the above commands, you should get an output that looks like this:

net eth1: Promiscuous mode disabled.
net eth1: Promiscuous mode disabled.
xilinx_axienet 41000000.ethernet eth1: Link is Down

Test the Ethernet ports

To test the Ethernet ports, we’ll need a PC with it’s own gigabit Ethernet port. Here I’m using my laptop which runs on Windows 10.

1. On the test PC, configure the Ethernet port to use a fixed IP address of 192.168.1.10.
2. Use an Ethernet cable to connect port 0 of the Ethernet FMC to the test PC. You should see the following message in Putty:

xilinx_axienet 41000000.ethernet eth1: Link is Up - 1Gbps/Full - flow control off

1. First let’s try pinging from the PC to the ZedBoard. Open a command window on the test PC and type the command: ping 192.168.1.11
2. Now to ping in the reverse direction (ZedBoard to PC), you will probably need to disable the public firewall on your PC.
3. On the Linux command line in Putty, type the command: ping -I eth1 192.168.1.10

You will have to press Ctrl-C to stop pinging. Notice that we have to use the argument “-I eth1” from the ZedBoard as there are multiple ports we could possibly ping from. We can do the same ping test to verify the other ports.

Source code Git repository

The sources for re-generating this project automatically can be found on Github here: ZedBoard Multi-port Ethernet design

Boot files for the ZedBoard

If you want to try out my boot files for the ZedBoard, download them here:

• BOOT.BIN
• image.ub

If you run into problems going through these instructions, just write me a comment below.