You said you want to communicate to the device using ethernet using a custom protocol. Does this mean you do not intend to use TCP/IP? Or, do you intend to use TCP/IP but the messages you send are custom? In other words, what's custom, the protocol, or the messages? If it's the protocol, good luck in designing basically a replacement to UDP, TCP/IP, or a myriad of other protocols that communicate over ethernet. If you're planning to use TCP/IP with your own messages, then LabVIEW has TCP/IP functions that allow you to send and receive messages to devices.
Oh, and a MAC address is just a unique identifier for the network interface of your device. If you intend to use TCP/IP you need an IP address which you can assign statically or get it dynamically if your network has a DHCP server and your device has a DCHP client.
I would suggest googling for a primer into network communications before delving too much into the actual programming.
