I have had the same problem with VLC refusing to play DVDs on a new Fedora 26 install.
I had already installed libdvdcss, vlc, and all the many codecs I routinely use.
VLC first refused to read the dvd. I then added myself to the cdrom group and it appeared to be trying to play but was giving errors (run from terminal) that seemed related to video drivers. The window was blinking and moving so fast it was almost impossible to click on the close button.
I had already disabled wayland and am running the xorg video manager since wayland causes mouse hangs/delays.
Then I found the reference earlier in this thread to the approach used by sparky68 on the videolan forum (posted here by robertdaleweir)
I found vlc was set to automatic by default and I switched it to XCB as suggested (I chose the "X11 Video Output (XCB)" instead of the "XVideo Output (XCB)" although both work for me. The DVDs now play properly.
Note 1: my PC uses the nouveau drivers.
Note 2: VLC was previously working perfectly in the same machine with my Fedora 25 install.
Note 3: HandBrake is installed and working as well as VLC -- both with the latest releases for F-26